3.2.1 -> 4.2 Upgrade Issue

[Jim Novy]

Jim Novy [http://community.zenoss.org/people/libjim] created the discussion

"3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/67951#67951

--------------------------------------------------------------
I've been running 3.2.1 for a while and am trying to upgrade to the 4.2.0 rpm on a fully updated CentOS 5 box.  I managed to get all the required software installed or upgraded and ran the yum localinstall on the rpm as the Installation Guide instructed, but when I issue the 'service zenoss start' command it runs through a large portion of the upgrade process and then hangs on the following step:

INFO:zen.migrate:Loading Reports

I've tried hitting ctrl-C to break out and restarting it, but it hangs at the same spot.  I also tried to restore the system from a backup and run through the process a second time, but am getting the same results.

Any suggestions?  Thanks!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/67951#67951]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/67967#67967

--------------------------------------------------------------
Could try provding the output a few lines preceeding this as well just to provide some context?

Also does top or ps indicate much or any activity on any specific process. I'm wondering if its just a slow process. I have not attempted an in place upgrade myself yet, but maybe with some additional info someone else might see something they hit before
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/67967#67967]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jim Novy]

Jim Novy [http://community.zenoss.org/people/libjim] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/67952#67952

--------------------------------------------------------------
Certainly!  Top shows no activity and I've let it hang there for 12 hours or better.  Here's a larger snippet of the process log:

|   -  Copying transaction 7179 of 71792012-08-17 15:27:13,803 [relstorage] INFO Copied tid 259127846033293704, | 1 records |  1.595 MB/s (  7179/  7179,100.00%) |
2012-08-17 15:27:13,803 [relstorage] INFO All 7179 transactions copied successfully in  1.3 minutes.


Removing ZEO from startup script...
-------------------------------------------------------------------------------
Done.

Converting Zenoss configuration files..
-------------------------------------------------------------------------------
Zope (/opt/zenoss/etc/zope.conf - UPDATED)

ZenActions (/opt/zenoss/etc/zenactions.conf - UPDATED)
ZenBackup (/opt/zenoss/etc/zenbackup.conf - UPDATED)
ZenRestore (/opt/zenoss/etc/zenrestore.conf - UPDATED)
ZenHub (/opt/zenoss/etc/zenhub.conf - UPDATED)
ZenJobs (/opt/zenoss/etc/zenjobs.conf - UPDATED)
ZenMib (/opt/zenoss/etc/zenmib.conf - UPDATED)
ZenMigrate (/opt/zenoss/etc/zenmigrate.conf - UPDATED)
ZenPack (/opt/zenoss/etc/zenpack.conf - UPDATED)
Global (/opt/zenoss/etc/global.conf - UPDATED)
adding default () for 'zodb-admin-password' in /opt/zenoss/etc/global.conf
adding default () for 'zep-admin-password' in /opt/zenoss/etc/global.conf
Adding RabbitMQ user: zenoss
Adding RabbitMQ vhost: /zenoss
Setting RabbitMQ permissions for user: zenoss
Creating database: zenoss_zep
Applying schema version: 1
Applying schema version: 2
Applying schema version: 3
Applying schema version: 4
Dropping database: zodb_session
Creating database: zodb_session
Applying schema version: 1
Upgrading to use persistent sessions...
/opt/zenoss/etc/zope.conf has been upgraded to use persistent sessions.
Starting zeneventserver...Migrating data...
INFO:zen.migrate:Database going to version Zenoss 4.2.0
INFO:zen.migrate:Installing CollectorIpv6Commands (4.0.0)
INFO:zen.migrate:Installing CreateMissingGuids (4.0.0)
INFO:zen.migrate:Installing DeleteNotificationSchedulesReport (4.0.0)
INFO:zen.migrate:Installing DeviceSearchCatalogUpdate (4.0.0)
INFO:zen.migrate:Installing FixHighLoadThreshold (4.0.0)
INFO:zen.migrate:Installing FixPerfFilesystemTransform (4.0.0)
INFO:zen.migrate:Installing JobsDelete_4_0 (4.0.0)
INFO:zen.migrate:Removing old job records
INFO:zen.migrate:Removing old job relationship
INFO:zen.migrate:Installing NotificationSubscriptions (4.0.0)
INFO:zen.migrate:Installing RemoveCommandsEditPermission (4.0.0)
INFO:zen.migrate:Installing RemoveEventMenus (4.0.0)
INFO:zen.migrate:Installing SessionAuthentication (4.0.0)
INFO:zen.migrate:Installing ToManyPersistentList (4.0.0)
INFO:zen.migrate:Installing Triggers (4.0.0)
INFO:zen.migrate:Installing addIpv6Network (4.0.0)
INFO:zen.migrate:Installing addzSnmpCollectionInterval (4.0.0)
INFO:zen.migrate:Installing snmpv3trap (4.0.0)
INFO:zen.migrate:Installing AddTriggerNotificationsForCommands (4.0.0)
INFO:zen.migrate:Installing AddTriggersAndNotifications (4.0.0)
INFO:zen.migrate:Done processing rule: rule1.
INFO:zen.migrate:Done processing rule: Library Down.
INFO:zen.migrate:Installing IpSearchCatalogUpdate (4.0.0)
INFO:zen.migrate:Updating Ipv4 Catalog
INFO:zen.migrate:rate=83.63/sec count=200
INFO:zen.migrate:Finished total time=2.70 rate=91.51 count=247
INFO:zen.migrate:Updating Ipv6 Catalog
INFO:zen.migrate:Finished total time=0.00 rate=0.00 count=0
INFO:zen.migrate:Installing RebuildPathIndex (4.0.0)
INFO:zen.migrate:path has already been reindexed
INFO:zen.migrate:Installing MigrateWindowCatalog (4.0.1)
INFO:zen.migrate:Installing FixLinkUpDownTransforms (4.0.70)
INFO:zen.migrate:Installing AddDiscardedEventsRate (4.1.2)
INFO:zen.migrate:Installing RemoveCycleTimeGraph (4.1.70)
INFO:zen.migrate:Installing AddUserInterfaceSettings (4.2.0)
INFO:zen.migrate:Installing ChangeSnmpDefaults (4.2.0)
INFO:zen.migrate:Installing ClearUserSettings (4.2.0)
INFO:zen.migrate:Installing DeviceSearchIpAddressCatalogCleanup (4.2.0)
INFO:zen.migrate:Installing FileSystemIndexName (4.2.0)
INFO:zen.migrate:Installing JobsDelete (4.2.0)
INFO:zen.migrate:Removing old job records
INFO:zen.migrate:Removing old job relationship
INFO:zen.migrate:Installing RemoveDCSHFromCommand (4.2.0)
INFO:zen.migrate:Installing RemoveDeviceZenMenus (4.2.0)
INFO:zen.migrate:Installing RemoveModificationsMenuItems (4.2.0)
INFO:zen.migrate:Installing SetIgnoreParamsOnZenDaemons (4.2.0)
INFO:zen.migrate:Installing SnmpWalkWithPort (4.2.0)
INFO:zen.migrate:Installing addIgnoreParametersWhenModeling (4.2.0)
INFO:zen.migrate:Adding ignoreParametersWhenModeling to all process classes.
INFO:zen.migrate:Installing addzInterfaceMapIgnoreDescriptions (4.2.0)
INFO:zen.migrate:Installing discoveryMaxOidsPerRequest (4.2.0)
INFO:zen.migrate:Setting zMaxOIDPerRequest to 10 on /Discovered
INFO:zen.migrate:Installing fixEmailNotificationClearSubjectFormat (4.2.0)
INFO:zen.migrate:Setting default values for E-mail notifications.
INFO:zen.migrate:Installing ifOperStatusEthernetCsmacd64 (4.2.0)
INFO:zen.migrate:Installing ifOperStatusRRDCreateCommand (4.2.0)
WARNING:zen.migrate:
Changing the definition for ifOperStatus. Please
remove all rrd files with this name: ifOperStatus_ifOperStatus.rrd.

You can remove all the rrd files with the following command as the zenoss user:
find $ZENHOME -name "ifOperStatus_ifOperStatus.rrd" -delete
INFO:zen.migrate:Installing moveZepConfig (4.2.0)
INFO:zen.migrate:Moving parameters from zeneventserver.conf to zenoss_zep config table.
INFO:zen.migrate:Installing perDeviceCompCatalog (4.2.0)
INFO:zen.migrate:Installing removeClearHeartbeatsMenuItem (4.2.0)
INFO:zen.migrate:Installing setDefaultsOnEmailNotifications (4.2.0)
INFO:zen.migrate:Setting default values for E-mail notifications.
INFO:zen.migrate:Installing zRemoveAutoAllocate (4.2.0)
INFO:zen.migrate:Installing zenmodelerModeledDevices (4.2.0)
INFO:zen.migrate:Adding modeledDevices datapoint and graph.
INFO:zen.migrate:Installing ifOperStatusEvents (4.2.0)
INFO:zen.migrate:Loading Reports
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/67952#67952]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/67969#67969

--------------------------------------------------------------
Only 12 hours, impatient much :) J/K of course lol. I'm half thinking this might warrant a bug report on jira.zenoss.com. If there is a problem the upgrade process should let you know I would assume.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/67969#67969]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/67953#67953

--------------------------------------------------------------
This is not an unusual amount of time for the v4.2 migration. One of my largest DC's took several days to finish migrating. :)

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/67953#67953]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68244#68244

--------------------------------------------------------------
If any other upgrade program does not display any activity to the console or gui for 10-20 minutes, I believe that there is something not working. If this is the expected behavior of ZenOss, it is _NOT_ good. The migration took time, but it showed output of a process on the screen. When it "hangs" on the Loading Reports.. it feels like FAIL... why do we see this way of not catch errors all the time.

Btw: I also see the hanging on "Loading Reports". I would not buy this product as long as I see these small problems getting its way into zenoss.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68244#68244]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68265#68265

--------------------------------------------------------------
rofls:

I agree, some aspects of the upgrade don't really indicate to the user that all is well. In many cases I waited long periods to find, when it finally did continue, that zenmigrate stubbed its toe on something and I have to do it again. The time for the upgrade the where it hangs can be heavily dependant on the hardware the migration is running on and the individual instance. What size is your installation and what sort of hardware are you running on?

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68265#68265]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68276#68276

--------------------------------------------------------------
The hardware on the test machine is not that great, but what I'm interested in is to getting past the "Loading Reports".. either by knowing that I need to wait or to find out what's wrong..

I'm about to check if the zenmigrate -v 10 helps. (Founc in the bin/install-functions.sh after upgrade)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68276#68276]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68271#68271

--------------------------------------------------------------
rolfs:

It sounds like it might be waiting for mysql or relstorage to catch up. Depending on the setup this could be normal. What sort of CPU and memory resources are available? I'm interested what steps are listed for 4.2 if you run zenmigrate --list.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68271#68271]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68279#68279

--------------------------------------------------------------
Loading Reports are after all the zenmigrate steps and it's not listed as a seperate step in zenmigrate. I'm letting the init.d/zenoss script invoke zenmigrate, just with the extra parameter -v 10 for the debug output.

Haven't checked what these does commands are doing in Products/ZenModel/migrate/Migrate.py yet:
        if not self.options.steps:
            self.message('Loading Reports')
            rl = ReportLoader(noopts=True, app=self.app)
            rl.options.logseverity = self.options.logseverity + 10
            rl.setupLogging()
            rl.loadDatabase()
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68279#68279]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68308#68308

--------------------------------------------------------------
with a -v 10 on the zenmigrate, it gives me this info:

DEBUG:zen.migrate:removed 718 .pyc files from Products
INFO:zen.migrate:Loading Reports
DEBUG:zen.GlobalConfig:Unknown option: --zodb-admin-user
DEBUG:zen.GlobalConfig:Unknown option: --zodb-admin-password
DEBUG:zen.GlobalConfig:Unknown option: --amqphost
DEBUG:zen.GlobalConfig:Unknown option: --amqpport
DEBUG:zen.GlobalConfig:Unknown option: --amqpvhost
DEBUG:zen.GlobalConfig:Unknown option: --amqpuser
DEBUG:zen.GlobalConfig:Unknown option: --amqppassword
DEBUG:zen.GlobalConfig:Unknown option: --amqpusessl
DEBUG:zen.GlobalConfig:Unknown option: --amqpadminport
DEBUG:zen.GlobalConfig:Unknown option: --amqpadminusessl
DEBUG:zen.GlobalConfig:Unknown option: --zep-db-type
DEBUG:zen.GlobalConfig:Unknown option: --zep-host
DEBUG:zen.GlobalConfig:Unknown option: --zep-port
DEBUG:zen.GlobalConfig:Unknown option: --zep-db
DEBUG:zen.GlobalConfig:Unknown option: --zep-user
DEBUG:zen.GlobalConfig:Unknown option: --zep-password
DEBUG:zen.GlobalConfig:Unknown option: --zep-admin-user
DEBUG:zen.GlobalConfig:Unknown option: --zep-admin-password
DEBUG:zen.GlobalConfig:Unknown option: --zep-uri
DEBUG:zen.GlobalConfig:Unknown option: --zodb-db-type
2012-08-30 08:36:27,958 INFO zen.ReportLoader: Loading reports from /opt/zenoss/Products/ZenReports/reports
2012-08-30 08:36:27,973 INFO zen.ReportLoader: loading: /Device Reports/MAC Addresses

And there it stopped almost 2 hours ago.. Any ideas?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68308#68308]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68340#68340

--------------------------------------------------------------
I've tried to add and remove the ZenPacks.sodonnell.MACReport to see if it could cleanup the zodb or make a difference. I think that the MACReport was installed once back in the days..

I found a MAC_Addresses.rpt file which I now will try to delete before the upgrade:
./Products/ZenReports/reports/Device_Reports/MAC_Addresses.rpt

Regards,
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68340#68340]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68407#68407

--------------------------------------------------------------
rolfs:

I dont really think that rpt will cause the problem. Were you able to move forward? What sort of hardware are you doing the migration on?

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68407#68407]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68417#68417

--------------------------------------------------------------
6GB ram, x86_64, CentOS release 5.5 (Final), Xeon(TM) CPU 3.00GHz, 4 core on my test-setup.

Looking at the process when it seems to have stopped shows little signs of action. No real cpu use. The strace does not even show much of activity for the zenmigrate process.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68417#68417]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68418#68418

--------------------------------------------------------------
I've added more debug lines to the code:

DEBUG:zen.migrate:removed 715 .pyc files from Products
INFO:zen.migrate:Loading Reports - skipping...
INFO:zen.migrate: self.success()
INFO:zen.migrate: got into success()...
INFO:zen.migrate:Committing changes
DEBUG:zen.queuepublisher:beforeCompletionHook on tx <transaction._transaction.Transaction object at 0x230a1590>
DEBUG:zen.queuepublisher:discarded 1 messages of 1719 total
DEBUG:zen.queuepublisher:ModelEventList batch size 1717
DEBUG:zen.zenoss.protocols.amqp:getChannel() in lib/python/zenoss/protocols/amqp.py
DEBUG:zen.zenoss.protocols.amqp:Connection localhost:5672, zenoss, zenoss, /zenoss
DEBUG:zen.zenoss.protocols.amqp:Connecting to RabbitMQ...
DEBUG:zen.zenoss.protocols.amqp:not self._channel
DEBUG:zen.zenoss.protocols.amqp:return out of getChannel() in python/zenoss/protocols/amqp.py
DEBUG:zen.zenoss.protocols.amqp:getChannel() in lib/python/zenoss/protocols/amqp.py
DEBUG:zen.zenoss.protocols.amqp:return out of getChannel() in python/zenoss/protocols/amqp.py
DEBUG:zen.zenoss.protocols.amqp:getChannel() in lib/python/zenoss/protocols/amqp.py
DEBUG:zen.zenoss.protocols.amqp:return out of getChannel() in python/zenoss/protocols/amqp.py
DEBUG:zen.protocols:Creating exchange: zenoss.events.modelchange
DEBUG:zen.zenoss.protocols.amqp:Publishing with routing key zenoss.event.modelchange to exchange zenoss.events.modelchange

I think it stops at this command:
channel.basic_publish(msg, exchangeConfig.name, routing_key, mandatory=mandatory, immediate=immediate)
which I think is running the code below from lib/python/amqplib/client_0_8/channel.py:

args = AMQPWriter()
if ticket is not None:
    args.write_short(ticket)
else:
    args.write_short(self.default_ticket)
args.write_shortstr(exchange)
args.write_shortstr(routing_key)
args.write_bit(mandatory)
args.write_bit(immediate)

self._send_method((60, 40), args, msg)



The rabbitmq logfile shows a connection:
==> /var/log/rabbitmq/***@servername.log <==

=INFO REPORT==== 6-Sep-2012::11:38:17 ===
accepting AMQP connection <0.30257.1> (127.0.0.1:55428 -> 127.0.0.1:5672)


So.. question becomes: how to debug rabbitmq to see what's happening in the queueing?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68418#68418]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/68432#68432

--------------------------------------------------------------
rolfs:

What's your mysql conf look like? Can you show engine innodb status? I'm thinking there's a deadlock during commit.

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68432#68432]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rolfs]

rolfs [http://community.zenoss.org/people/rolfs] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/73138#73138

--------------------------------------------------------------
Hi!

This had nothing todo with my mysql configuration, but the fact that my /var partition only had 500MB free.

My rabbitmq logfile shows that it has to little disk space:

/var/log/rabbitmq/***@zenoss.log
=INFO REPORT==== 8-May-2013::15:17:31 ===
Disk free space insufficient. Free bytes:474038272 Limit:1000000000

Doing a disk lvm extend made the "Loading Reports" continue

[***@zenoss rabbitmq]# /sbin/lvm lvextend -L5G /dev/vg0/var
  Extending logical volume var to 5.00 GB
  Logical volume var successfully resized
[***@zenoss rabbitmq]# /sbin/resize2fs -p /dev/vg0/var
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg0/var is mounted on /var; on-line resizing required
Performing an on-line resize of /dev/vg0/var to 1310720 (4k) blocks.
The filesystem on /dev/vg0/var is now 1310720 blocks long.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73138#73138]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/73141#73141

--------------------------------------------------------------
rolfs:

What version of rabbit are you running?

--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73141#73141]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: 3.2.1 -> 4.2 Upgrade Issue"

To view the discussion, visit: http://community.zenoss.org/message/73143#73143

--------------------------------------------------------------
rolfts:

How much disk space is on '/'? What's rabbitmqctl list_connections as root return?

Best,
--Shane Scott (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73143#73143]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]