Zenoss 4.2 Zenhub is down.

[Spencer Krum]

Spencer Krum [http://community.zenoss.org/people/nibz] created the discussion

"Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68847#68847

--------------------------------------------------------------
Hello,

I have Centos 6.3 and a fresh run of the core-autodeploy-4.2.sh script but zenoss is still not working. Init scripts all fire. Website comes up. But loggin it doesn't work. Nothing in Z2.log. Problem appears to be zenhub. Zenhubs log insists that zenhub is on and working but zensyslog seems to thing zenhub doesn't respond.

zenhub.log
`
created_time: 1349258579174
event_class: "/App/Start"
actor {
  element_type_id: DEVICE
  element_identifier: "localhost"
  element_sub_type_id: COMPONENT
  element_sub_identifier: "zenhub"
}
summary: "zenhub started"
severity: SEVERITY_CLEAR
event_key: ""
, with this routing key: zenoss.zenevent.app.start
2012-10-03 03:02:59,187 DEBUG zen.zenoss.protocols.amqp: Connecting to RabbitMQ...
2012-10-03 03:02:59,188 DEBUG zen.protocols: Creating exchange: zenoss.zenevents.raw
2012-10-03 03:02:59,188 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.zenevent.app.start to exchange zenoss.zenevents.raw
2012-10-03 03:03:00,656 DEBUG zen.ZenHub: Registered 4 invalidation filters.
2012-10-03 03:03:00,729 DEBUG zen.thresholds: Updating threshold ('high event queue', ('localhost collector', ''))
2012-10-03 03:03:00,730 DEBUG zen.thresholds: Updating threshold ('zenmodeler cycle time', ('localhost collector', ''))
2012-10-03 03:03:00,730 DEBUG zen.ZenHub: Starting /opt/zenoss/bin/zenhubworker run -C /opt/zenoss/var/zenhub/localhost_worker.conf
2012-10-03 03:03:00,741 DEBUG zen.ZenHub: Starting /opt/zenoss/bin/zenhubworker run -C /opt/zenoss/var/zenhub/localhost_worker.conf
2012-10-03 03:03:00,749 DEBUG zen.Events: ===============  incoming event  ===============
2012-10-03 03:03:00,757 DEBUG zen.Events: Got a localhost zenhub heartbeat event (timeout 90 sec).
2012-10-03 03:03:00,769 DEBUG zen.zenoss.protocols.amqp: Connecting to RabbitMQ...
2012-10-03 03:03:00,770 DEBUG zen.protocols: Creating exchange: zenoss.heartbeats
2012-10-03 03:03:00,770 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.heartbeat.localhost to exchange zenoss.heartbeats
2012-10-03 03:05:39,478 DEBUG zen.Plugins: Loading collector plugins from: /opt/zenoss/Products/DataCollector/plugins
2012-10-03 03:05:39,481 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090
2012-10-03 03:05:39,485 DEBUG zen.Events: ===============  incoming event  ===============
2012-10-03 03:05:39,487 DEBUG zen.queuepublisher: About to publish this event to the raw event queue:uuid: "e246b026-0d41-11e2-882c-ccaa77f68540"
created_time: 1349258739485
event_class: "/App/Start"
actor {
  element_type_id: DEVICE
  element_identifier: "localhost"
  element_sub_type_id: COMPONENT
  element_sub_identifier: "zenhub"
}
summary: "zenhub started"
severity: SEVERITY_CLEAR
event_key: ""
, with this routing key: zenoss.zenevent.app.start
2012-10-03 03:05:39,514 DEBUG zen.zenoss.protocols.amqp: Connecting to RabbitMQ...
2012-10-03 03:05:39,515 DEBUG zen.protocols: Creating exchange: zenoss.zenevents.raw
2012-10-03 03:05:39,515 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.zenevent.app.start to exchange zenoss.zenevents.raw
2012-10-03 03:05:40,459 DEBUG zen.ZenHub: Registered 4 invalidation filters.
2012-10-03 03:05:40,498 DEBUG zen.thresholds: Updating threshold ('high event queue', ('localhost collector', ''))
2012-10-03 03:05:40,499 DEBUG zen.thresholds: Updating threshold ('zenmodeler cycle time', ('localhost collector', ''))
2012-10-03 03:05:40,499 DEBUG zen.ZenHub: Starting /opt/zenoss/bin/zenhubworker run -C /opt/zenoss/var/zenhub/localhost_worker.conf
2012-10-03 03:05:40,504 DEBUG zen.ZenHub: Starting /opt/zenoss/bin/zenhubworker run -C /opt/zenoss/var/zenhub/localhost_worker.conf
2012-10-03 03:05:40,513 DEBUG zen.Events: ===============  incoming event  ===============
2012-10-03 03:05:40,514 DEBUG zen.Events: Got a localhost zenhub heartbeat event (timeout 90 sec).
2012-10-03 03:05:40,526 DEBUG zen.zenoss.protocols.amqp: Connecting to RabbitMQ...
2012-10-03 03:05:40,529 DEBUG zen.protocols: Creating exchange: zenoss.heartbeats
2012-10-03 03:05:40,529 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.heartbeat.localhost to exchange zenoss.heartbeats




zensyslog:


2012-10-03 03:33:45,183 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 03:33:45,183 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 03:33:45,183 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 03:33:52,896 INFO zen.zensyslog: Attempting to connect to zenhub
2012-10-03 03:34:52,897 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds
2012-10-03 03:38:45,186 INFO zen.maintenance: Performing periodic maintenance
2012-10-03 03:38:45,187 INFO zen.zensyslog: Counter eventCount, value 5
2012-10-03 03:38:45,188 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2012-10-03 03:38:45,188 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 03:38:45,188 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 03:38:45,189 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 03:40:12,463 INFO zen.zensyslog: Attempting to connect to zenhub
2012-10-03 03:41:12,472 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds
2012-10-03 03:43:45,192 INFO zen.maintenance: Performing periodic maintenance
2012-10-03 03:43:45,192 INFO zen.zensyslog: Counter eventCount, value 5
2012-10-03 03:43:45,193 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2012-10-03 03:43:45,193 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 03:43:45,194 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 03:43:45,194 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 03:45:19,731 INFO zen.zensyslog: Attempting to connect to zenhub
2012-10-03 03:46:19,736 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds
2012-10-03 03:48:45,197 INFO zen.maintenance: Performing periodic maintenance
2012-10-03 03:48:45,197 INFO zen.zensyslog: Counter eventCount, value 5
2012-10-03 03:48:45,198 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2012-10-03 03:48:45,199 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 03:48:45,199 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 03:48:45,199 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 03:51:37,073 INFO zen.zensyslog: Attempting to connect to zenhub
2012-10-03 03:52:37,078 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds
2012-10-03 03:53:45,202 INFO zen.maintenance: Performing periodic maintenance
2012-10-03 03:53:45,203 INFO zen.zensyslog: Counter eventCount, value 5
2012-10-03 03:53:45,204 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2012-10-03 03:53:45,204 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 03:53:45,204 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 03:53:45,205 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 03:57:09,399 INFO zen.zensyslog: Attempting to connect to zenhub
2012-10-03 03:58:09,400 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds
2012-10-03 03:58:45,207 INFO zen.maintenance: Performing periodic maintenance
2012-10-03 03:58:45,208 INFO zen.zensyslog: Counter eventCount, value 5
2012-10-03 03:58:45,209 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2012-10-03 03:58:45,209 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 03:58:45,209 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 03:58:45,210 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 04:02:46,127 INFO zen.zensyslog: Attempting to connect to zenhub
2012-10-03 04:03:45,345 INFO zen.maintenance: Performing periodic maintenance
2012-10-03 04:03:45,345 INFO zen.zensyslog: Counter eventCount, value 5
2012-10-03 04:03:45,346 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2012-10-03 04:03:45,346 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 04:03:45,347 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 04:03:45,347 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 04:03:46,128 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds
2012-10-03 04:07:57,194 INFO zen.zensyslog: Attempting to connect to zenhub
2012-10-03 04:08:45,350 INFO zen.maintenance: Performing periodic maintenance
2012-10-03 04:08:45,351 INFO zen.zensyslog: Counter eventCount, value 5
2012-10-03 04:08:45,352 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2012-10-03 04:08:45,352 INFO zen.collector.scheduler: Tasks: 0 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-03 04:08:45,352 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2012-10-03 04:08:45,353 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-03 04:08:57,198 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds


Is anyone else seeing this with 4.2? I'm happy to run any commands or get more logs for people. Thanks.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68847#68847]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Spencer Krum]

Spencer Krum [http://community.zenoss.org/people/nibz] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68878#68878

--------------------------------------------------------------
I added that to .bashrc and .bash_profile then ran /opt/zenoss/bin/zenctl restart. Zenctl returned successully but tailing /opt/zenoss/log/zensyslog still reports zenhub is down.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68878#68878]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68905#68905

--------------------------------------------------------------
Spencer:

Are any errors reported in the zeneventserver.log?

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68905#68905]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Ben S]

Ben S [http://community.zenoss.org/people/bin] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68841#68841

--------------------------------------------------------------
Same here, Centos 6.3 and using the autodeploy script = Same problem.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68841#68841]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Michael Fabricius]

Michael Fabricius [http://community.zenoss.org/people/micsame] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68842#68842

--------------------------------------------------------------
Same issue here,

I have downloaded zenoss 4.2 as a VMware player and the OS that are used are CentOS 5.4 so maybe that is the version we should use.

[***@localhost ~]# cat /etc/redhat-release
CentOS release 5.4 (Final
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68842#68842]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68863#68863

--------------------------------------------------------------
Did you see this:
go to the message on Zenoss Community (http://community.zenoss.org/message/68815#68815) ?

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68863#68863]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ALyarskiy]

ALyarskiy [http://community.zenoss.org/people/ALyarskiy] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68974#68974

--------------------------------------------------------------
Same problem on ScientificLinux 6.2 - 6.3, CentOS 6.3 only on xen domU, tried on KVM - all ok.
After install zenhub running but with enabled debug there are messages about all workers busy and job queue grows.

OS:
[***@zenoss ~]# cat /etc/redhat-release
Scientific Linux release 6.2 (Carbon)
[***@zenoss ~]# uname -a
Linux zenoss 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11 CST 2011 x86_64 x86_64 x86_64 GNU/Linux

zenprocess.log:
2012-10-10 11:21:42,234 INFO zen.zenprocess: Connecting to localhost:8789
2012-10-10 11:22:12,239 ERROR zen.zenprocess: Timeout connecting to zenhub: is it running?
2012-10-10 11:22:12,240 INFO zen.maintenance: Performing periodic maintenance
2012-10-10 11:22:12,242 INFO zen.zenprocess: 0 devices processed (0 datapoints)
2012-10-10 11:22:12,242 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2012-10-10 11:22:12,243 WARNING zen.zenprocess: No service named 'EventService': ZenHub may be disconnected
2012-10-10 11:22:12,243 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down
2012-10-10 11:22:12,244 ERROR zen.collector.config: Task configLoader configure failed: ZenHub is down


zenhub.log (with debug):
2012-10-10 11:23:06,585 DEBUG zen.hub: adding listener for localhost:EventService
2012-10-10 11:23:06,587 DEBUG zen.hub: adding listener for localhost:PingPerformanceConfig
2012-10-10 11:23:06,596 DEBUG zen.ZenHub: worklist has 1 items
2012-10-10 11:23:06,596 DEBUG zen.ZenHub: get candidate workers for sendEvents...
2012-10-10 11:23:06,596 DEBUG zen.ZenHub: candidate workers are [1]
2012-10-10 11:23:06,596 DEBUG zen.ZenHub: Giving sendEvents to worker 1, (localhost:Products.ZenHub.services.EventService.sendEvents)
2012-10-10 11:23:06,634 DEBUG zen.ZenHub: worklist has 1 items
2012-10-10 11:23:06,634 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:07,033 DEBUG zen.ZenHub: worklist has 1 items
2012-10-10 11:23:07,033 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:07,974 DEBUG zen.hub: adding listener for localhost:EventService
2012-10-10 11:23:07,977 DEBUG zen.hub: adding listener for localhost:ZenStatusConfig
2012-10-10 11:23:07,986 DEBUG zen.ZenHub: worklist has 2 items
2012-10-10 11:23:07,986 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:07,986 DEBUG zen.ZenHub: worklist has 3 items
2012-10-10 11:23:07,987 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:08,025 DEBUG zen.ZenHub: worklist has 4 items
2012-10-10 11:23:08,025 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:12,033 DEBUG zen.ZenHub: worklist has 4 items
2012-10-10 11:23:12,035 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:12,057 DEBUG zen.Events: ===============  incoming event  ===============
2012-10-10 11:23:12,057 DEBUG zen.Events: Got a localhost zenhub heartbeat event (timeout 90 sec).
2012-10-10 11:23:12,057 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.heartbeat.localhost to exchange zenoss.heartbeats
2012-10-10 11:23:17,035 DEBUG zen.ZenHub: worklist has 4 items
2012-10-10 11:23:17,036 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:22,036 DEBUG zen.ZenHub: worklist has 4 items
2012-10-10 11:23:22,037 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:23,854 DEBUG zen.hub: adding listener for localhost:EventService
2012-10-10 11:23:23,863 DEBUG zen.hub: adding listener for localhost:ModelerService
2012-10-10 11:23:23,868 DEBUG zen.ZenHub: worklist has 5 items
2012-10-10 11:23:23,868 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:23,870 DEBUG zen.ZenHub: worklist has 6 items
2012-10-10 11:23:23,870 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:25,751 DEBUG zen.ZenHub: worklist has 7 items
2012-10-10 11:23:25,751 DEBUG zen.ZenHub: all workers are busy
2012-10-10 11:23:27,038 DEBUG zen.ZenHub: worklist has 7 items
2012-10-10 11:23:27,039 DEBUG zen.ZenHub: all workers are busy

Strace of zenhub process in attachment. VM running and i can provide additional info.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68974#68974]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/68978#68978

--------------------------------------------------------------
Alyarskiy:

What are the specs of your VM and how many devices are you monitoring with zenoss? Can you post your zeneventserver.conf, zenhub.conf and zeneventd.conf? Are you using distributed collectors?

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68978#68978]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ALyarskiy]

ALyarskiy [http://community.zenoss.org/people/ALyarskiy] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/69016#69016

--------------------------------------------------------------
Default installation of zenoss 4.2, no monitored devices, default configs.

Xen domU, 4 VCPU, 2G RAM, 20Gb FC (Storwise 7000).

Only localhost as monitored device but it is not modeled.

zeneventserver.conf:
# The URI to the RabbitMQ server. This property is ignored if
# $ZENHOME/etc/global.conf can be read in which the AMQP URI will be
# built from the properties found in that location.
#zep.amqp.uri=amqp://zenoss:***@localhost:5672/zenoss


# The number of milliseconds to wait between connection retries
#zep.amqp.retry=60000


# The default maximum number of events returned in an event query
#zep.query.limit=1000


# The maximum number of minutes a closed event may not be seen again before moving to archive.
#zep.max_event_archive_interval_minutes=43200


# JDBC Protocol
# Loaded from global.conf zep_db_type first, then falls back to the definition in this file.
#zep.jdbc.protocol=mysql


# JDBC Hostname
# Loaded from global.conf zep_host first, then falls back to the definition in this file.
#zep.jdbc.hostname=localhost


# JDBC Port
# Loaded from global.conf zep_port first, then falls back to the definition in this file.
#zep.jdbc.port=3306


# JDBC Database Name
# Loaded from global.conf zep_db first, then falls back to the definition in this file.
#zep.jdbc.dbname=zenoss_zep


# JDBC Username
# Loaded from global.conf zep_user first, then falls back to the definition in this file.
#zep.jdbc.username=zenoss


# JDBC Password
# Loaded from global.conf zep_password first, then falls back to the definition in this file.
#zep.jdbc.password=zenoss


# The default auto commit state of pooled connections
#zep.jdbc.pool.default_auto_commit=false


# The default read only state of pooled connections
#zep.jdbc.pool.default_read_only=false


# The default transaction isolation of pooled connections
#zep.jdbc.pool.default_transaction_isolation=2


# The maximum number of active connections in the pool
#zep.jdbc.pool.max_active=10


# The maximum number of idle connections in the pool
#zep.jdbc.pool.max_idle=10


# The minimum number of idle connections in the pool
#zep.jdbc.pool.min_idle=3


# The initial number of connections in the pool
#zep.jdbc.pool.initial_size=3


# The maximum amount of time to wait for a connection from the pool
#zep.jdbc.pool.max_wait=-1


# Whether to validate connections when borrowing from the pool
#zep.jdbc.pool.test_on_borrow=false


# Whether to validate connections when returning to the pool
#zep.jdbc.pool.test_on_return=false


# Whether to validation connections periodically while idle
#zep.jdbc.pool.test_while_idle=false


# Amount of time between idle connection validation, abandoned cleaner, and
# idle pool resizing.
#zep.jdbc.pool.time_between_eviction_runs_millis=5000


# Amount of time before a connection is considered idle
#zep.jdbc.pool.min_evictable_idle_time_millis=60000


# Interceptors to run
#zep.jdbc.pool.jdbc_interceptors=ConnectionState


# How often to validate connections in the pool
#zep.jdbc.pool.validation_interval=30000


# Whether JMX can be used to configure the pool
#zep.jdbc.pool.jmx_enabled=true


# True if a fair queue is used by the connection pool.
#zep.jdbc.pool.fair_queue=true


# Timed out connections will get closed when this percentage of the pool is full.
#zep.jdbc.pool.abandon_when_percentage_full=0


# Time in milliseconds to keep connections alive.
#zep.jdbc.pool.max_age=0


# Set to true if ProxyConnection class should use String.equals instead of ==
# when comparing method names.
#zep.jdbc.pool.use_equals=true


# Whether to remove abandoned connections if they exceed the
# remove_abandoned_timeout.
#zep.jdbc.pool.remove_abandoned=false


# The amount of time (in seconds) before a connection is considered abandoned.
#zep.jdbc.pool.remove_abandoned_timeout=60


# Whether to log stack traces of abandoned connections.
#zep.jdbc.pool.log_abandoned=false


# Instead of abandoning connections, logs a warning if log_abandoned is true.
# If this value is <= 0, then no suspect checking will be performed.
#zep.jdbc.pool.suspect_timeout=0


# The default number of retries performed when a database deadlock is encountered.
#zep.jdbc.deadlock_retries=5


# The core pool size used by the primary executor service.
# By default, this is set to the number of processors in the system.
#zep.executor.core_pool_size=4


# The maximum pool size used by the primary executor service.
# By default, this is set to the number of processors in the system.
#zep.executor.max_pool_size=4


# The maximum number of entries waiting to execute on the executor service.
#zep.executor.queue_capacity=10


# Index Directory
#zep.index.dir=var/zeneventserver/index


# Index RAM buffer size
#zep.index.ram_buffer_size_mb=16.0


# Heartbeat check interval in seconds
#zep.heartbeat.interval_seconds=60


# Database optimization is performed on this interval (in minutes). Specify an interval <= 0 to disable.
#zep.database.optimize_minutes=60


## Comma separated list of ZEP plug-ins to disable.
#zep.plugins.disabled=


## Whether to disable all external ZEP plug-ins (from ZenPacks).
#zep.plugins.external.disabled=false


## Optional configuration settings for each plug-in
#plugin.<ID>.<NAME>=<VALUE>


## The default number of trigger rule compilations to cache.
#plugin.TriggerPlugin.triggerRuleCacheSize=200


## Partitioning parameters for partitioned tables in ZEP.
##
## Values are:
##
##   partition.<table>.unit = The TimeUnit of each partition.
##   partition.<table>.duration = The duration of each partition.
##   partition.<table>.initial_past_partitions = The initial number of partitions to create in the past.
##   partition.<table>.future_partitions = The number of future partitions to maintain.
##


#partition.event_archive.unit=DAYS
#partition.event_archive.duration=1
# We want to configure the maximum number of initial partitions for migration.
#partition.event_archive.initial_past_partitions=90
#partition.event_archive.future_partitions=3


#partition.event_time.unit=HOURS
#partition.event_time.duration=1
# We want to configure the maximum number of initial partitions for migration.
#partition.event_time.initial_past_partitions=0
#partition.event_time.future_partitions=2

zenhub.conf:
#
# Configuration file for zenhub
#
#  To enable a particular option, uncomment the desired entry.
#
# Parameter     Setting
# ---------     -------
#
# Logging severity threshold, default: 20
#logseverity 20
#
# Override the default logging path
#logpath None
#
# Max size of log file in KB; default
#  10240, default: 10240
#maxlogsize 10240
#
# Max number of back up log files; default
#  3, default: 3
#maxbackuplogs 3
#
# Use an alternate configuration file
#configfile None
#
# Generate an XML file containing command-line
#  switches., default: False
#genxmlconfigs False
#
# User to become when running default:zenoss,
#  default: zenoss
#uid zenoss
#
# Cycle continuously on cycleInterval
#  from Zope, default: False
#cycle False
#
# Launch into the background, default: False
#daemon False
#
# Log to console and log file, default: False
#duallog False
#
# output log info in HTML table format,
#  default: False
#weblog False
#
# Run under a supervisor which will restart
#  it, default: False
#watchdog False
#
# The path to the watchdog reporting socket
#watchdogPath None
#
# Wait seconds for initial heartbeat
#starttimeout None
#
# Set listener socket options.For option
#  details: man 7 socket, default: []
#socketOption []
#
# Port to use for XML-based Remote Procedure
#  Calls (RPC), default: 8081
#xmlrpcport 8081
#
# Port to use for Twisted's pb service,
#  default: 8789
#pbport 8789
#
# File where passwords are stored, default:
#  /Users/pkw/Projects/zenoss/trunk/home/etc/hubpasswd
#passwd /Users/pkw/Projects/zenoss/trunk/home/etc/hubpasswd
#
# Name of the distributed monitor this
#  hub runs on, default: localhost
#monitor localhost
#
# Number of worker instances to handle
#  requests, default: 2
#workers 2
#
# Run higher priority jobs before lower
#  priority ones, default: False
#prioritize False
#
# Allow any priority job to run on any
#  worker, default: False
#anyworker False
#
# Log current worker state to $ZENHOME/log/workerstats,
#  default: False
#logworkerstats False
#
# root object for data load (i.e. /zport/dmd),
#  default: /zport/dmd
#zodb-dataroot /zport/dmd
#
# in memory cachesize default: 1000, default:
#  1000
#zodb-cachesize 1000
#
# hostname of the MySQL server for ZODB,
#  default: localhost
#zodb-host localhost
#
# port of the MySQL server for ZODB, default:
#  3306
#zodb-port 3306
#
# user of the MySQL server for ZODB, default:
#  zenoss
#zodb-user zenoss
#
# password of the MySQL server for ZODB,
#  default: zenoss
#zodb-password zenoss
#
# Name of database for MySQL object store,
#  default: zodb
#zodb-db zodb
#
# Name of socket file for MySQL server
#  connection if host is localhost
#zodb-socket None
#
# memcached servers to use for object
#  cache (eg. 127.0.0.1:11211), default:
#  127.0.0.1:11211
#zodb-cacheservers 127.0.0.1:11211
#
# Defer polling the database for the specified
#  maximum time interval, in seconds.
#  This will default to 60 only if --zodb-cacheservers
#  is set.
#zodb-poll-interval None
#

zeneventd.conf:
#
# Configuration file for zeneventd
#
#  To enable a particular option, uncomment the desired entry.
#
# Parameter     Setting
# ---------     -------
#
# Logging severity threshold, default: 20
#logseverity 20
#
# Override the default logging path
#logpath None
#
# Max size of log file in KB; default
#  10240, default: 10240
#maxlogsize 10240
#
# Max number of back up log files; default
#  3, default: 3
#maxbackuplogs 3
#
# Use an alternate configuration file
#configfile None
#
# Generate an XML file containing command-line
#  switches., default: False
#genxmlconfigs False
#
# User to become when running default:zenoss,
#  default: zenoss
#uid zenoss
#
# Cycle continuously on cycleInterval
#  from Zope, default: False
#cycle False
#
# Launch into the background, default: False
#daemon False
#
# Log to console and log file, default: False
#duallog False
#
# output log info in HTML table format,
#  default: False
#weblog False
#
# Run under a supervisor which will restart
#  it, default: False
#watchdog False
#
# The path to the watchdog reporting socket
#watchdogPath None
#
# Wait seconds for initial heartbeat
#starttimeout None
#
# Set listener socket options.For option
#  details: man 7 socket, default: []
#socketOption []
#
# Cycle, in seconds, for maintenance tasks.
#  Default is 60., default: 60
#maintenancecycle 60
#
# Force sync() before processing every
#  event; default is to sync() no more
#  often than once every 1/2 second.,
#  default: False
#synceveryevent False
#
# root object for data load (i.e. /zport/dmd),
#  default: /zport/dmd
#zodb-dataroot /zport/dmd
#
# in memory cachesize default: 1000, default:
#  1000
#zodb-cachesize 1000
#
# hostname of the MySQL server for ZODB,
#  default: localhost
#zodb-host localhost
#
# port of the MySQL server for ZODB, default:
#  3306
#zodb-port 3306
#
# user of the MySQL server for ZODB, default:
#  zenoss
#zodb-user zenoss
#
# password of the MySQL server for ZODB,
#  default: zenoss
#zodb-password zenoss
#
# Name of database for MySQL object store,
#  default: zodb
#zodb-db zodb
#
# Name of socket file for MySQL server
#  connection if host is localhost
#zodb-socket None
#
# memcached servers to use for object
#  cache (eg. 127.0.0.1:11211), default:
#  127.0.0.1:11211
#zodb-cacheservers 127.0.0.1:11211
#
# Defer polling the database for the specified
#  maximum time interval, in seconds.
#  This will default to 60 only if --zodb-cacheservers
#  is set.
#zodb-poll-interval None
#

I do not using distributed collectors or something else - i am just trying to install it with basic setup =)))
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69016#69016]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/69022#69022

--------------------------------------------------------------
Alyarskiy:

As root execute /usr/sbin/rabbitmqctl list_queues -p /zenoss. I'm wondering where the bottle neck is.

Best,
--Shane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69022#69022]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ALyarskiy]

ALyarskiy [http://community.zenoss.org/people/ALyarskiy] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/69044#69044

--------------------------------------------------------------
Nothing there...
[***@zenoss ~]# /usr/sbin/rabbitmqctl list_queues -p /zenoss
Listing queues ...
celery          0
zenoss.queues.zep.modelchange          0
zenoss.queues.zep.signal          0
zenoss.queues.zep.migrated.summary          0
zenoss.queues.zep.rawevents          0
zenoss.queues.zep.heartbeats          0
zenoss.queues.zep.zenevents          0
zenoss.celeryd.pidbox          0
zenoss.queues.zep.migrated.archive          0
...done.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69044#69044]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ALyarskiy]

ALyarskiy [http://community.zenoss.org/people/ALyarskiy] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/69045#69045

--------------------------------------------------------------
How can i debug zenhubworkers? It seems zenhub detects status of the workers incorrectly:

2012-10-12 10:37:17,571 INFO zen.ZenHub: Worklist Stats:
        Events: 6
        Other:  13
        ApplyDataMaps:  0
        Total:  19
Worker Stats:
        0:Busy  [Busy localhost:Products.ZenHub.services.EventService.sendEvents Idle:0.073s]   2984.784s
        1:Busy  [Busy localhost:Products.ZenHub.services.EventService.sendEvents Idle:1.100s]   2980.573s
        2:Busy  [Busy localhost:Products.ZenHub.services.EventService.sendEvents]       170075.409s

But:

1421 ?        S      0:03 /opt/zenoss/bin/python /opt/zenoss/Products/ZenHub/zenhubworker.py --configfile /opt/zenoss/etc/zenhubworker.conf -C /opt/zenoss/var/zenhub/localhost_worker.conf
2100 ?        S      0:03 /opt/zenoss/bin/python /opt/zenoss/Products/ZenHub/zenhubworker.py --configfile /opt/zenoss/etc/zenhubworker.conf -C /opt/zenoss/var/zenhub/localhost_worker.conf
2101 ?        S      0:04 /opt/zenoss/bin/python /opt/zenoss/Products/ZenHub/zenhubworker.py --configfile /opt/zenoss/etc/zenhubworker.conf -C /opt/zenoss/var/zenhub/localhost_worker.conf

Seems something locking worker process. No IOWait on system.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69045#69045]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Ellery Blank]

Ellery Blank [http://community.zenoss.org/people/etendards] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/69374#69374

--------------------------------------------------------------
I am having a similar issue with ZenHub.  Has this issue been addressed with 4.2?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69374#69374]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/69450#69450

--------------------------------------------------------------
Alyarskiy:

I have a suspiscion that zenoss is corrupt. Would it be possible for you to try again using the most recent VMware appliance or most recent auto deploy script?

Ellery:

Can you supply more details?

Thanks!

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69450#69450]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Thomas Vogel]

Thomas Vogel [http://community.zenoss.org/people/evoget] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/69629#69629

--------------------------------------------------------------
I had the same problem with my 4.2 on rhel6.
My problem was the rabbitmq-server. The rabbitmq-server blocks by default every connection, if the memory is over 40% or the filesystem hasn't min. 1gb free space.
Check the following as root:

rabbitmqctl list_connections

If there is something with "blocked" or "blocking" you have the same problem.
You can modify this limitations:

http://www.rabbitmq.com/memory.html http://www.rabbitmq.com/memory.html
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69629#69629]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Re: Zenoss 4.2 Zenhub is down."

To view the discussion, visit: http://community.zenoss.org/message/71231#71231

--------------------------------------------------------------
You, my friend, are my hero!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71231#71231]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]