Thomas Bjorseth
2012-10-16 09:23:36 UTC
Thomas Bjorseth [http://community.zenoss.org/people/popeye70] created the discussion
"zenhub dies seconds after starting"
To view the discussion, visit: http://community.zenoss.org/message/69239#69239
--------------------------------------------------------------
Hello all.
I have an old 3.2.1 instance that I have been toying around with for a while, testing Zenoss and monitoring a few devices (35-40 servers). Recently I was ready to move into production state, and wanted to upgrade to the latest Zenoss version. I decided to do a fresh start to get away from the old stack installer that I was using, and wasn't planning on migrating any settings - the old system was ready to be ditched completely. Hence, yesterday I installed a CentOS 6.3 basic server and went through the autodeploy-process. Within minutes, my shiny, new Zenoss Core 4.2.0 instance was up and running.
For a couple of hours, I hardened CentOS a bit (enabling firewall, tightening ssh) and started adding devices to Zenoss. Everything was working perfectly, but then I noticed that the system clock was two hours off, and I startet NTP to get an accurate clock. I also renamed the server (as it was moved from preprod to production). I am not sure if this is the reason for the problems that followed, but thought I should mention it.
Now, after a system reboot and/or Zenoss restart, zenhub appears to be starting, then dies. All other Zenoss processes start as expected. In the zenhub.log, I get the following entries:
+2012-10-16 10:43:45,330 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090+
+2012-10-16 10:43:48,342 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
+2012-10-16 10:43:51,346 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
+2012-10-16 10:43:51,346 ERROR zen.ZenHub: Unable to send an event+
+Traceback (most recent call last):+
+Â File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent+
+Â Â Â self.zem.sendEvent(Event(**kw))+
+Â File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent+
+Â Â Â event = self._publishEvent(event)+
+Â File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent+
+Â Â Â publisher.publish(event)+
+Â File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish+
+Â Â Â self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)+
+Â File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish+
+Â Â Â mandatory, immediate)+
+Â File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish+
+Â Â Â headers=headers, declareExchange=declareExchange)+
+Â File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish+
+Â Â Â raise Exception("Could not publish message. Connection may be down")+
+Exception: Could not publish message. Connection may be down+
+2012-10-16 10:43:57,255 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
+2012-10-16 10:44:00,259 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
I can log in to the web GUI, and I can see all my devices, but no errors are logged, and no graphs are graphed. I can no longer add anything to the system (no users or devices), and no changes to the devices/groups/systems/locations hierarchy can be made. Basically, I have a shiny new system that tells me that nothing is wrong as no events are received or handled, but clearly something is very wrong.
Does Zenoss store the hostname and/or IP address in any of it's config files or the database, causing these problems when changing name/IP?
Does any of you know where I should start looking, to avoid spending hours looking in the wrong place? Instead of spending a workday or two trying blindly to fix the problem, I might as well reinstall CentOS and Zenoss, but that wouldn't teach me anything and wouldn't be any fun.
Thanks in advance! :)
--------------------------------------------------------------
Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69239#69239]
Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
"zenhub dies seconds after starting"
To view the discussion, visit: http://community.zenoss.org/message/69239#69239
--------------------------------------------------------------
Hello all.
I have an old 3.2.1 instance that I have been toying around with for a while, testing Zenoss and monitoring a few devices (35-40 servers). Recently I was ready to move into production state, and wanted to upgrade to the latest Zenoss version. I decided to do a fresh start to get away from the old stack installer that I was using, and wasn't planning on migrating any settings - the old system was ready to be ditched completely. Hence, yesterday I installed a CentOS 6.3 basic server and went through the autodeploy-process. Within minutes, my shiny, new Zenoss Core 4.2.0 instance was up and running.
For a couple of hours, I hardened CentOS a bit (enabling firewall, tightening ssh) and started adding devices to Zenoss. Everything was working perfectly, but then I noticed that the system clock was two hours off, and I startet NTP to get an accurate clock. I also renamed the server (as it was moved from preprod to production). I am not sure if this is the reason for the problems that followed, but thought I should mention it.
Now, after a system reboot and/or Zenoss restart, zenhub appears to be starting, then dies. All other Zenoss processes start as expected. In the zenhub.log, I get the following entries:
+2012-10-16 10:43:45,330 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090+
+2012-10-16 10:43:48,342 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
+2012-10-16 10:43:51,346 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
+2012-10-16 10:43:51,346 ERROR zen.ZenHub: Unable to send an event+
+Traceback (most recent call last):+
+Â File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent+
+Â Â Â self.zem.sendEvent(Event(**kw))+
+Â File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent+
+Â Â Â event = self._publishEvent(event)+
+Â File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent+
+Â Â Â publisher.publish(event)+
+Â File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish+
+Â Â Â self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)+
+Â File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish+
+Â Â Â mandatory, immediate)+
+Â File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish+
+Â Â Â headers=headers, declareExchange=declareExchange)+
+Â File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish+
+Â Â Â raise Exception("Could not publish message. Connection may be down")+
+Exception: Could not publish message. Connection may be down+
+2012-10-16 10:43:57,255 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
+2012-10-16 10:44:00,259 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer+
I can log in to the web GUI, and I can see all my devices, but no errors are logged, and no graphs are graphed. I can no longer add anything to the system (no users or devices), and no changes to the devices/groups/systems/locations hierarchy can be made. Basically, I have a shiny new system that tells me that nothing is wrong as no events are received or handled, but clearly something is very wrong.
Does Zenoss store the hostname and/or IP address in any of it's config files or the database, causing these problems when changing name/IP?
Does any of you know where I should start looking, to avoid spending hours looking in the wrong place? Instead of spending a workday or two trying blindly to fix the problem, I might as well reinstall CentOS and Zenoss, but that wouldn't teach me anything and wouldn't be any fun.
Thanks in advance! :)
--------------------------------------------------------------
Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69239#69239]
Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]