Can't keep zenhub running

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/67793#67793

--------------------------------------------------------------
I sure hope someone can help me figure this out.  Been through all the logs, forums, and anything else I can think of, so as a last resort, I'm reaching out the community.  I'm not a power linux user by any means, but  know enough to be dangerous, to give you some background.   All we need is simple monitoring and alerting, but without Zenhub, we don't get alerts, which isn't good.  Zenhub won't stay running for a more than 2-3 days at this point, and it's been that way for months.  There's nothing in the zenhub log.  It just stops.  I found this in zencommand log, which is the most information I can find on the issue.

2012-08-10 21:35:52,849 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 704, in doFetchConfig
    self.setPropertyItems(driver.next())
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
HubDown: ZenHub is down
2012-08-10 21:35:53,342 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 752, in start
    driver.next()
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
HubDown: ZenHub is down
2012-08-11 03:35:53,484 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 704, in doFetchConfig
    self.setPropertyItems(driver.next())
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
HubDown: ZenHub is down
2012-08-11 03:35:53,685 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 752, in start
    driver.next()
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex

We are running this on Ubuntu Linux 12.04.1 and Zenoss is the latest version: Zenoss 3.2.1.

I'd truly appreciate some help in getting this fixed.  Please let me know if there is any other information I can provide.  Thank you in advance.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/67793#67793]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/67983#67983

--------------------------------------------------------------
sunnydave:

On the zenoss master su zenoss then zenhub stop. Once it's stopped, ps aux | grep zenhub and kill any remaning workers if there are any. Once done, zenhub start -v10. Keep an eye on the $ZENHOME/log/zenhub.log. Post the log when zenhub dies. Can you also please post your $ZENHOME/etc/zenhub.conf?

Thanks!

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/67983#67983]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68128#68128

--------------------------------------------------------------
Hi Shane,

First off, really appreciate the response and apologize for the delay in getting back.  I been on the road, traveling for work.

I've done the steps above and restarted zenhub with verbose level 10 debugging (-v10).  I'll keep an eye out and post the log next time it fails.

As for the config of $ZENHOME/etc/zenhub.conf, not much to it, it appears, with the one line in the conf file commented out.

| #PARAMETER  | VALUE |

I'll post again when I have the logs after zenhub quits.  Thank you again for your time and assistance.  Best.
-Dave
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68128#68128]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68130#68130

--------------------------------------------------------------
sunnydave:

Anytime.

I await the log. :) Thanks man!

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68130#68130]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68172#68172

--------------------------------------------------------------
Hi Shane,

Zenhub quit at some point over the weekend, and I've captured the logs and downloaded the files.  However, the log collection was quite large.  I've got three (3) files of 10 mb each, and another, which is the latest, of about 5 mb.

I looked through them briefly to see if I could find where the daemon quitting occurred, but honestly, nothing jumped out at me.  How would you like me to get you the logs?  Thank you again for your assistance.

Dave
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68172#68172]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68173#68173

--------------------------------------------------------------
Hi Shane,

Additionally, I found this in zencommand.log.  In the last few months in trying to track this down, I've looked through all the logs, and this one references zenhub quitting with an error.  Not sure if it's related, but just so you have all the information, as well as a potential time stamp of the problem occurring.

2012-08-26 03:01:47,676 INFO zen.zencommand: config: <Products.ZenRRD.zencommand.DeviceConfig instance at 0x40aca28>
2012-08-26 07:59:38,657 ERROR zen.zencommand: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 587, in processSchedule
    c.start(self.pool).addBoth(self.finished)
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 342, in start
    d = pr.start(self)
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 92, in start
    reactor.spawnProcess(self, shell, self.cmdline, env=None)
  File "/usr/local/zenoss/python/lib/python2.6/site-packages/Twisted-8.1.0-py2.6-linux-x86_64.egg/twisted/internet/posixbase.py", line 224, in spawnProcess
    processProtocol, uid, gid, childFDs)
  File "/usr/local/zenoss/python/lib/python2.6/site-packages/Twisted-8.1.0-py2.6-linux-x86_64.egg/twisted/internet/process.py", line 538, in __init__
    self._fork(path, uid, gid, executable, args, environment, fdmap=fdmap)
  File "/usr/local/zenoss/python/lib/python2.6/site-packages/Twisted-8.1.0-py2.6-linux-x86_64.egg/twisted/internet/process.py", line 373, in _fork
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
2012-08-26 09:10:49,799 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 704, in doFetchConfig
    self.setPropertyItems(driver.next())
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
HubDown: ZenHub is down
2012-08-26 09:10:50,500 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 752, in start
    driver.next()
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
HubDown: ZenHub is down
2012-08-26 15:10:50,796 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 704, in doFetchConfig
    self.setPropertyItems(driver.next())
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
HubDown: ZenHub is down
2012-08-26 15:10:51,229 ERROR zen.zencommand: ZenHub is down
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 752, in start
    driver.next()
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
HubDown: ZenHub is down

Thank you.

Dave
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68173#68173]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68176#68176

--------------------------------------------------------------
sunnydave:

Can you email them to me at mailto:***@gmail.com ***@gmail.com?

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68176#68176]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68181#68181

--------------------------------------------------------------
Hi Shane,

I zipped up the files and sent them your way.  Thank you again.  I look forward to hearing back from you.  Best regards.

Dave
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68181#68181]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68182#68182

--------------------------------------------------------------
Oh no! Somebody dump sacked my inbox! :p

Just kidding. I got your logs sunnydave. I'll take a look and get back to you.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68182#68182]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chuck Hudock]

Chuck Hudock [http://community.zenoss.org/people/cjhudock] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68210#68210

--------------------------------------------------------------
Hello Gentlemen,
First of all I’m a newbie to Zenoss.   I downloaded the 4.2 VM image and loaded on an ESX server. After loading a few devices and a few MIB, I too am seeing my zenhub crash. It will not stay up at all. It immediately shuts down.

When I attempt to restart the service I get the following in the zenhub log:

2012-08-28 09:14:01,227 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090
2012-08-28 09:14:04,239 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-08-28 09:14:07,242 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-08-28 09:14:07,242 ERROR zen.ZenHub: Unable to send an event
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent
    self.zem.sendEvent(Event(**kw))
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent
    event = self._publishEvent(event)
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent
    publisher.publish(event)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish
    self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish
    mandatory, immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish
    headers=headers, declareExchange=declareExchange)
  File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish
    raise Exception("Could not publish message. Connection may be down")
Exception: Could not publish message. Connection may be down
2012-08-28 09:14:12,730 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-08-28 09:14:15,734 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

I deleted all of the zenhubworker processes with the same results.

Any idea?

Chuck
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68210#68210]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68215#68215

--------------------------------------------------------------
Chuck:

Looks like your rabbitmq-server daemon might not be running. Check that and restart zenhub.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68215#68215]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chuck Hudock]

Chuck Hudock [http://community.zenoss.org/people/cjhudock] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68198#68198

--------------------------------------------------------------
Shane,



The rabbitmg-server appears to be running. Performed a restart and checked
status.



Deleted zenhubworkers process and performed a start from the
Zenoss:Daemons. Same result.



Chuck
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68198#68198]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68217#68217

--------------------------------------------------------------
Chuck:

Interesting. I would double check that mysqld is running as well. Next Double check that $ZENHOMe/etc/globals.conf points to the local rabbit & mysqld instance with valid creds.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68217#68217]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68268#68268

--------------------------------------------------------------
Hi Shane,

I had our sysadmin load up the 4.2 virtual appliance yesterday on our esx host.  Firgured if we can't get the 3.2.1 version figured, perhaps we can at least migrate upwards for some stability.  I configured it basic with an IP, dns, timezone, local snmp, etc., and also installed sendmail MTA for notifications.  I didn't add any nodes at all for monitoring.  I did remove an EC2 instance that was pre-populated.  Once I rebooted the machine, I'm getting the same error as Chuck.  Zenhub won't start at all.

I looked at the globals.conf, and it all appears correct.  I certainly didn't make any changes there.  Everything is running (including rabbitmg), mysqld, etc.

2012-08-29 10:49:24,997 ERROR zen.ZenHub: Unable to send an event
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent
    self.zem.sendEvent(Event(**kw))
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent
    event = self._publishEvent(event)
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent
    publisher.publish(event)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish
    self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish
    mandatory, immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish
    headers=headers, declareExchange=declareExchange)
  File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish
    raise Exception("Could not publish message. Connection may be down")
Exception: Could not publish message. Connection may be down
2012-08-29 10:49:47,766 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-08-29 10:49:50,775 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

That's the latest on my end.  Thanks again for your time and assistance.

Dave
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68268#68268]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68269#68269

--------------------------------------------------------------
This sounds alot like http://community.zenoss.org/message/67159#67159 http://community.zenoss.org/message/67159#67159. Have you looked at creating the rabbitmq-env.conf file?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68269#68269]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68270#68270

--------------------------------------------------------------
Dave:

Looking at the zenhub logs, nothing sticks out. Whats the memory load on this box look like?

For the VM Ware applicance, it's definitely a case where rabbit isn't accessible. As root can you run '/usr/sbin/rabbitmqctl list_queues -p /zenoss' and post the result? If the bin isnt found, mlocate rabbitmqctl and use that path to run it. You could also try running $ZENHOME/bin/zenoss_upgrade_pre as the root user to have zenoss go through its usual upgrade steps which include configuring rabbits exchanges. I don't know if that'll fix the problem on the VM Ware product, however.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68270#68270]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68272#68272

--------------------------------------------------------------
Here's the memory load of the 3.2.1 box.

***@blahblan-zenoss1:~# free m
             total       used       free     shared    buffers     cached
Mem:       1019512     940764      78748          0       1924      30032
-/+ buffers/cache:     908808     110704
Swap:      1044476    1010912      33564

Regarding the 4.2 box.

I just loaded up a rabbitmq-env.conf file here and added my variables:
/etc/rabbitmq
[***@blah-zenoss2 rabbitmq]# more rabbitmq-env.conf
#example rabbitmq-env.conf file entries
#Rename the node
NODENAME=blah-zenoss2.fqdn.com
#Config file location and new filename bunnies.config
CONFIG_FILE=/etc/rabbitmq/rabbitmq-env.conf

I then rebooted the box, but am seeing the same error and zenhub won't start.

Here's the output you requestd.  I replaced our fqdn with blah

[***@blahblah-zenoss2 ~]# /usr/sbin/rabbitmqctl list_queues -p /zenoss
Listing queues ...
Error: unable to connect to node 'blah-***@blah-zenoss2': nodedown

DIAGNOSTICS
===========

nodes in question: ['blah-***@blah-zenoss2']

hosts, their running nodes and ports:
- blah-zenoss2: [{rabbitmqctl5414,51346}]

current node details:
- node name: '***@blah-zenoss2'
- home dir: /var/lib/rabbitmq
- cookie hash: Gzyq/2JMhTgLWXJQrb+UxA==

I did add the hostname and IP in /etc/hosts.  Thank you.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68272#68272]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chuck Hudock]

Chuck Hudock [http://community.zenoss.org/people/cjhudock] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68285#68285

--------------------------------------------------------------
I have found the VMware snapshot function extremely helpful.  But I have no
solution to the problem.  If I modify the config files  such as
/etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0  the
system becomes unstable. I believe I should use some utility to make the
changes in a more structured way.



I ended up adding a reservation in the DHCP server to have a consistent IP
address and not touching the network files at all.



Chuck
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68285#68285]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68324#68324

--------------------------------------------------------------
Hi all,

I got rabbitmq to run on the 4.2 box by messing with some hostname settings in /etc/hosts and /etc/sysconfig/network and the rabbitmq-env.conf file referenced above, but I'm still not able to run zenhub.  I get the same error in the zenhub logs.

2012-08-30 12:23:25,919 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090
2012-08-30 12:23:28,957 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-08-30 12:23:31,976 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-08-30 12:23:31,977 ERROR zen.ZenHub: Unable to send an event
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent
    self.zem.sendEvent(Event(**kw))
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent
    event = self._publishEvent(event)
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent
    publisher.publish(event)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish
    self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish
    mandatory, immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish
    headers=headers, declareExchange=declareExchange)
  File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish
    raise Exception("Could not publish message. Connection may be down")
Exception: Could not publish message. Connection may be down
2012-08-30 12:23:39,092 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-08-30 12:23:42,096 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

Here is the rabbitmq output to show that it is in fact running.

[***@zenoss2 ~]# /usr/sbin/rabbitmqctl list_queues -p /zenoss
Listing queues ...
...done.
[***@zenoss2 ~]#
[***@zenoss2 ~]#
[***@zenoss2 ~]# rabbitmqctl -n zenoss2  status
Status of node ***@zenoss2 ...
[{pid,3462},
{running_applications,[{rabbit,"RabbitMQ","2.8.4"},
                        {os_mon,"CPO  CXC 138 46","2.1.8"},
                        {sasl,"SASL  CXC 138 11","2.1.5.4"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.7"},
                        {stdlib,"ERTS  CXC 138 10","1.15.5"},
                        {kernel,"ERTS  CXC 138 10","2.12.5"}]},
{os,{unix,linux}},
{erlang_version,"Erlang (BEAM) emulator version 5.6.5 [source] [64-bit] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,[{total,24980344},
          {processes,10049144},
          {processes_used,10038448},
          {system,14931200},
          {atom,1069665},
          {atom_used,1053972},
          {binary,137952},
          {code,11791567},
          {ets,745336}]},
{vm_memory_high_watermark,0.39999999990517116},
{vm_memory_limit,843625267},
{disk_free_limit,1000000000},
{disk_free,5531357184},
{file_descriptors,[{total_limit,924},
                    {total_used,6},
                    {sockets_limit,829},
                    {sockets_used,4}]},
{processes,[{limit,1048576},{used,127}]},
{run_queue,0},
{uptime,220}]
...done.

Appreciate all the assistance.

Dave
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68324#68324]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68342#68342

--------------------------------------------------------------

Post by sunnydave
Listing queues ...
celery  0
zenoss.queues.zep.signal        0
zenoss.queues.zep.modelchange   0
zenoss.queues.zep.migrated.summary      0
zenoss.queues.zep.rawevents     0
zenoss.queues.zep.heartbeats    0
zenoss.queues.zep.zenevents     0
localhost.localdomain.celeryd.pidbox    0
zenoss.queues.zep.migrated.archive      0
...done.

What does the output of netstat look like in regards to rabbitmq?

Post by sunnydave
netstat -ano | grep 5672

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68342#68342]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68344#68344

--------------------------------------------------------------
Can you run the following commands and provide the output?

rabbitmqctl list_users
cat /opt/zenoss/etc/global.conf | grep amqp

Dump the permissions for the currently configured amqpuser

amqp_user=`cat /opt/zenoss/etc/global.conf | grep amqpuser | gawk '{print $2}'`
rabbitmqctl list_user_permissions -p /zenoss $amqp_user

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68344#68344]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68408#68408

--------------------------------------------------------------
sunnydave:

Hum, something is seriously wrong. Have you tried running $ZENHOME/bin/zenoss_upgrade_pre as the root user? This will go through all the upgrade steps including rebuilding the rabbit queues and configuring zenoss to talk to them.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68408#68408]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[sunnydave]

sunnydave [http://community.zenoss.org/people/sunnydave] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68488#68488

--------------------------------------------------------------
Hi Shane and dpetzel,

First off, my apologies for dropping off the radar.  I was traveling for a work conference/summit last week and was out of touch entirely after the holiday weekend.  Second, a sincere thank you for your input and assistance.  I'm digging out today, and wanted to update you both on the latest.

Regarding my 3.2.1 build, it _appears_ it was a memory issue.  I asked the admin's of our ESX environment if I could up the RAM of the virtual box to 4 gigs, up from 1 gig.  Once that was done and the box was rebooted, zenhub has been running ever since.  It's been up for 11 days now.  Not that I'm out of the water, but I haven't had it up for more than 3 days for the last 4 months or so.  Feels like progress and the functionality I've been used to for the last 4 years.

Second, regarding the 4.2 build, I ended up starting again with a new virtual machine and will utilize the snapshot feature as I build/make changes to the box.  I had already deleted the box before your posts dpetzel, so I'm sorry not to provide feedback on your helpful input.  Very much appreciated and best regards.

Dave
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68488#68488]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68485#68485

--------------------------------------------------------------
Dave:

No problem, we all get busy. :)

Glad to hear it's working now! I had a feeling it might be resource related. Keep an eye on it and happy ZenOSSing!

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68485#68485]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Mark Walker]

Mark Walker [http://community.zenoss.org/people/BillyBob] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68796#68796

--------------------------------------------------------------
Shane,

I'm seeing this same problem with zenhub after reconfiguring the CentOS VM demo to run on a bridged LAN in VMware Workstation.


The demo VM was configured to use VMnet1 (host-only), I moved to VMnet0 (bridged) and altered the files network, ifcfg-eth0, hosts. Zenoss starts up and talks at the expected static address, but configuring scanning jobs fails silently.

When I check the log files I find zenhub has exited:

2012-09-28 11:15:05,924 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090
2012-09-28 11:15:09,041 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-09-28 11:15:12,046 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-09-28 11:15:12,047 ERROR zen.ZenHub: Unable to send an event
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent
    self.zem.sendEvent(Event(**kw))
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent
    event = self._publishEvent(event)
  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent
    publisher.publish(event)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish
    self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish
    mandatory, immediate)
  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish
    headers=headers, declareExchange=declareExchange)
  File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish
    raise Exception("Could not publish message. Connection may be down")
Exception: Could not publish message. Connection may be down
2012-09-28 11:15:18,005 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer
2012-09-28 11:15:21,009 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

When I attempt $ZENHOME/bin/zenoss_upgrade_pre as root:

[***@zenoss bin]# $ZENHOME/bin/zenoss_upgrade_pre
-bash: /bin/zenoss_upgrade_pre: No such file or directory
[***@zenoss bin]# pwd
/opt/zenoss/bin
[***@zenoss bin]# ./zenoss_upgrade_pre
./zenoss_upgrade_pre: line 32: /bin/zenoss_startup_config: No such file or directory
./zenoss_upgrade_pre: line 35: /bin/shared-functions.sh: No such file or directory
./zenoss_upgrade_pre: line 36: /bin/install-functions.sh: No such file or directory
./zenoss_upgrade_pre: line 39: confirm_jre: command not found
./zenoss_upgrade_pre: line 42: confirm_rrd: command not found
./zenoss_upgrade_pre: line 45: shebang: command not found
./zenoss_upgrade_pre: line 49: update_conf_files: command not found
grep: /etc/zope.conf: No such file or directory
./zenoss_upgrade_pre: line 54: create_zodb_db: command not found
-bash: /bin/python: No such file or directory
./zenoss_upgrade_pre: line 62: upgrade_conf_options_42: command not found
./zenoss_upgrade_pre: line 65: /bin/zenoss_startup_config: No such file or directory
sed: can't read /etc/zenstatus.conf: No such file or directory
./zenoss_upgrade_pre: line 79: start_rabbitmq: command not found
./zenoss_upgrade_pre: line 82: configure_amqp: command not found
./zenoss_upgrade_pre: line 85: /bin/zenoss_upgrade_patch_zenpacks.sh: No such file or directory
./zenoss_upgrade_pre: line 88: create_zep_db: command not found
./zenoss_upgrade_pre: line 93: remove_zep_jdbc_config: command not found
./zenoss_upgrade_pre: line 97: create_zodb_session_db: command not found
./zenoss_upgrade_pre: line 100: patch_zopectl_and_runzope: command not found
./zenoss_upgrade_pre: line 103: upgrade_to_persistent_sessions: command not found
./zenoss_upgrade_pre: line 115: run_zenmigrate: command not found
[***@zenoss bin]#


Should I be doing zenoss_upgrade_pre as the zenoss user?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68796#68796]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68815#68815

--------------------------------------------------------------
Mark:

Sounds like paths didn't get setup.

In your /root/.bashrc add the following:

# User specific aliases and functions
export ZENHOME="/opt/zenoss"
export PATH="${ZENHOME}/bin:${PATH}"
export PYTHONPATH="/opt/zenoss/lib/python"
export LD_LIBRARY_PATH="/opt/zenoss/lib"

And make sure your /root/.bash_profile contains the following:

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

Once done exit the root user then re su root. echo $ZENHOME. If it returns /opt/zenoss then give the $ZENHOME/bin/zenoss_upgrade_pre again.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68815#68815]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68816#68816

--------------------------------------------------------------
Mark:

Anytime! :)

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68816#68816]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Mark Walker]

Mark Walker [http://community.zenoss.org/people/BillyBob] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/68797#68797

--------------------------------------------------------------
Shane,

The environment settings allowed the preconfig to work, and I can now create/submit jobs that run and
complete successfully.

Thank you for the assist.

Mark
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68797#68797]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[J C]

J C [http://community.zenoss.org/people/jcole] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/69057#69057

--------------------------------------------------------------
Thanks guys...this solved my problem with zenhub not running.

1. su - root
2. /etc/init.d/zenoss stop                                       # stop zenoss
3. cat << EOF >> .bashrc                                     # add ENV's
export ZENHOME="/opt/zenoss"
export PATH="${ZENHOME}/bin:${PATH}"
export PYTHONPATH="/opt/zenoss/lib/python"
export LD_LIBRARY_PATH="/opt/zenoss/lib"
EOF
4. /etc/init.d/zenoss start                                      # start zenoss
5. $ZENHOME/bin/zenctl xstatus |grep zenhub      # ck zenhub staus
Running        26316      zenhub

Nice to find a thread with an answer instead of just the questions...  :x

Excellent work zenmasters of zenoss!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69057#69057]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/69059#69059

--------------------------------------------------------------
JC:

Anytime.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69059#69059]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[suman suman]

suman suman [http://community.zenoss.org/people/greensuman] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/73870#73870

--------------------------------------------------------------
Shane,

After shut down  and start the zenoss server .it  stoped to monitor the events .the script zenoss_upgrade_pre
helped to recover.

Thanks
suman
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73870#73870]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Richard Carter]

Richard Carter [http://community.zenoss.org/people/daydreamblighter] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/69573#69573

--------------------------------------------------------------
Shane, this fixed my issue with zenhub on the 4.2 appliance too. Thanks very much!

Thread here: http://community.zenoss.org/thread/18651?tstart=0 http://community.zenoss.org/thread/18651?tstart=0
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69573#69573]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[szogu]

szogu [http://community.zenoss.org/people/szogu] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/70219#70219

--------------------------------------------------------------
I did all this in .bashrc, re-login (root) and run "$ZENHOME/bin/zenoss_upgrade_pre". Script stopped on "Dropping database: zodb_session" since 20 minutes (database was empty!).

Should I stop zenoss before?

Tommy
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70219#70219]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[J C]

J C [http://community.zenoss.org/people/jcole] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/70220#70220

--------------------------------------------------------------

   2)  /etc/init.d/zenoss stop                                       # stop zenoss



I did stop Zenoss prior to applying the script.   Give that a go...
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70220#70220]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/70257#70257

--------------------------------------------------------------
Richard:

Anytime! :)

szogu:

You'll need to stop zenoss first then run the zenoss_upgrade_pre as root after fixing the .bashrc and .bash_profile.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70257#70257]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[szogu]

szogu [http://community.zenoss.org/people/szogu] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/70269#70269

--------------------------------------------------------------
Hi Shane,
I'm back at work tonight.
I stopped zenoss and run zenoss_upgrade_pre after fixing .bashrc and .bash_profile

Result:
[***@zmonbm101 ~]# zenoss_upgrade_pre
Checking RRDtool version >= 1.4.7: [ OK ] 1.4.7
adding default () for 'zodb-admin-password' in /opt/zenoss/etc/global.conf
adding default () for 'zep-admin-password' in /opt/zenoss/etc/global.conf
Creating database: zenoss_zep
Dropping database: zodb_session
Creating database: zodb_session
Applying schema version: 1
Upgrading to use persistent sessions...
Already using persistent sessions or using custom session storage. zope.conf wil
Starting zeneventserver...zeneventserver failed to start within 180 seconds.
Please check the following log file for more details:
  /opt/zenoss/log/zeneventserver.log
Migrating data...
INFO:zen.migrate:Installing AddUserInterfaceSettings (4.2.0)
INFO:zen.migrate:Installing ChangeSnmpDefaults (4.2.0)
INFO:zen.migrate:Installing ClearUserSettings (4.2.0)
INFO:zen.migrate:Installing DeviceSearchIpAddressCatalogCleanup (4.2.0)
INFO:zen.migrate:Installing FileSystemIndexName (4.2.0)
INFO:zen.migrate:Installing JobsDelete (4.2.0)
INFO:zen.migrate:Removing old job records
INFO:zen.migrate:Removing old job relationship
INFO:zen.migrate:Installing RemoveDCSHFromCommand (4.2.0)
INFO:zen.migrate:Installing RemoveDeviceZenMenus (4.2.0)
INFO:zen.migrate:Installing RemoveModificationsMenuItems (4.2.0)
INFO:zen.migrate:Installing SetIgnoreParamsOnZenDaemons (4.2.0)
INFO:zen.migrate:Installing SnmpWalkWithPort (4.2.0)
INFO:zen.migrate:Installing addIgnoreParametersWhenModeling (4.2.0)
INFO:zen.migrate:Adding ignoreParametersWhenModeling to all process classes.
INFO:zen.migrate:Installing addzInterfaceMapIgnoreDescriptions (4.2.0)
INFO:zen.migrate:Installing discoveryMaxOidsPerRequest (4.2.0)
INFO:zen.migrate:Installing fixEmailNotificationClearSubjectFormat (4.2.0)
INFO:zen.migrate:Setting default values for E-mail notifications.
INFO:zen.migrate:Installing ifOperStatusEthernetCsmacd64 (4.2.0)
INFO:zen.migrate:Installing ifOperStatusRRDCreateCommand (4.2.0)
WARNING:zen.migrate:
Changing the definition for ifOperStatus. Please
remove all rrd files with this name: ifOperStatus_ifOperStatus.rrd.


You can remove all the rrd files with the following command as the zenoss user:
find $ZENHOME -name "ifOperStatus_ifOperStatus.rrd" -delete
INFO:zen.migrate:Installing moveZepConfig (4.2.0)
INFO:zen.migrate:Moving parameters from zeneventserver.conf to zenoss_zep config                                                                                                                                table.
INFO:zen.migrate:Installing perDeviceCompCatalog (4.2.0)
INFO:zen.migrate:Installing removeClearHeartbeatsMenuItem (4.2.0)
INFO:zen.migrate:Installing setDefaultsOnEmailNotifications (4.2.0)
INFO:zen.migrate:Setting default values for E-mail notifications.
INFO:zen.migrate:Installing zRemoveAutoAllocate (4.2.0)
INFO:zen.migrate:Installing zenmodelerModeledDevices (4.2.0)
INFO:zen.migrate:Installing ifOperStatusEvents (4.2.0)
INFO:zen.migrate:Loading Reports

As you can see "zeneventserver failed to start within 180 seconds".
but
[***@zmonbm101 ~]$ zenoss xstatus
Status        | PID     | Daemon
-------------------------------------------------------------------
Running        13978      zeneventserver
Stopped                   zopectl
Stopped                   zenhub
Stopped                   zenjobs
Stopped                   zeneventd
Stopped                   zenping
Stopped                   zensyslog
Stopped                   zenstatus
Stopped                   zenactiond
Stopped                   zentrap
Stopped                   zenmodeler
Stopped                   zenperfsnmp
Stopped                   zencommand
Stopped                   zenprocess
Stopped                   zenrrdcached
[***@zmonbm101 ~]$ zenoss start
Daemon: zeneventserver is already running
Daemon: zopectl .
daemon process started, pid=15476
Daemon: zenhub starting...
Daemon: zenjobs starting...
Daemon: zeneventd starting...
Daemon: zenping starting...
Daemon: zensyslog starting...
Daemon: zenstatus starting...
Daemon: zenactiond starting...
Daemon: zentrap starting...
Daemon: zenmodeler starting...
Daemon: zenperfsnmp starting...
Daemon: zencommand starting...
Daemon: zenprocess starting...
Daemon: zenrrdcached starting...

Unfortunately all deamons report:

2012-12-02 22:39:22,502 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-12-02 22:39:22,597 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-12-02 22:42:46,454 INFO zen.zensyslog: Attempting to connect to zenhub

I attached logs files.

CPU load is on normal level. I wrote about this here: http://community.zenoss.org/message/70242#70242#70242 http://community.zenoss.org/message/70242#70242

And waiting for response from the system is a little frustrating for me...
[***@zmonbm101 ~]$ date; zenoss xstatus; date
Sun Dec  2 22:15:58 CET 2012
Status        | PID     | Daemon
-------------------------------------------------------------------
Running        13978      zeneventserver
Running        15476      zopectl
Running        15565      zenhub
Running        15773      zenjobs
Running        15870      zeneventd
Running        15942      zenping
Running        16000      zensyslog
Running        16065      zenstatus
Running        16126      zenactiond
Running        16206      zentrap
Running        16263      zenmodeler
Running        16326      zenperfsnmp
Running        16390      zencommand
Running        16447      zenprocess
Running        16453      zenrrdcached
Sun Dec  2 22:17:37 CET 2012

Is this normal?

thanks in advance
Tommy
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70269#70269]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[J C]

J C [http://community.zenoss.org/people/jcole] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/70274#70274

--------------------------------------------------------------
Double check and make sure mysql is running...

Check out http://community.zenoss.org/message/67861#67861 http://community.zenoss.org/message/67861#67861.

There may be some threads there you can pull on for CentOS 6.3 / Zenoss 4.2.  
See the note regarding +"disabling SELINUX"+.

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70274#70274]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[suman suman]

suman suman [http://community.zenoss.org/people/greensuman] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/73868#73868

--------------------------------------------------------------
Shane ,

i followed the same steps .worked for me .
Thanks a lot

Thanks
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73868#73868]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/73869#73869

--------------------------------------------------------------
Anytime.

--Shane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73869#73869]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Emocke]

Emocke [http://community.zenoss.org/people/Emocke] created the discussion

"Re: Can't keep zenhub running"

To view the discussion, visit: http://community.zenoss.org/message/74593#74593

--------------------------------------------------------------
Hi Shane

Sorry to bug you about the same type of issue.
I have now tried everything you supplied and still my Zenbub is not runnning.
To give some background:
I had 4.2.3 and had a Gui error which most people(Forums) thought its linked to my JAVA being updated.After a Reboot my server was stuck on starting the JEXEC services.
Since i could not find any issue with the JAVA installed, and also not able to disable JEXEC, i decided to take a backup via SSH Command and then re-install.
My new install was 4.2.4
I had some issues linking Zenoss with ZENUP and decided to go without it.
The server booted up. I did not check all the services but assumed all working.
I then restored my backup. After all my devices was back i started receiving Link to path wrong.
I decided to look at the services and found ZENHUB not running.

So currently i have tried what you said. Checked my Rabbit and still no luck.

Do you think i Need to re-install and to use an earlier backup.

***@ZenossMain zenoss]# zenoss status
Daemon: zeneventserver program running; pid=32186
Daemon: zopectl program running; pid=32441
Daemon: zenrrdcached program running; pid=32446
Daemon: *zenhub not running*


*service zenoss status*
*rabbitmqctl list_queues -p /zenoss*
*rabbitmqctl add_user zenoss zenoss*
*rabbitmqctl add_vhost /zenoss*
*rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'*
*service zenoss restart*
*service rabbitmq-server status*
*service zenoss status*
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74593#74593]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]