can't delete some devices and/or processes

[yscotty]

|| Zenoss (http://www.zenoss.com/) | Zenoss 2.5.2 |
|| OS (http://www.tldp.org/) | Linux (x86_64) 2.6.32 (Linux mnp01zen2 2.6.32-24-server #39-Ubuntu SMP Wed Jul 28 06:21:40 UTC 2010 x86_64) |
|| Zope (http://www.zope.org/) | Zope 2.11.2 |
|| Python (http://www.python.org/) | Python 2.4.4 |
|| Database (http://www.mysql.com/) | MySQL 5.0.45 (Ver 5.0.45) |
|| RRD (http://oss.oetiker.ch/rrdtool) | RRDtool 1.3.8 |
|| Twisted (http://twistedmatrix.com/trac) | Twisted 8.1.0 |
|| NetSnmp (http://net-snmp.sourceforge.net/) | NetSnmp 5.4.1 |
|| PyNetSnmp (http://www.zenoss.com/) | PyNetSnmp 0.28.13 |
|| WMI (http://www.zenoss.com/) | Wmi 1.3.6 |
 
Originally, my problem started with certain locations devices reporting as down, when they clearly were not. I was able to just restart zenping to get things working normally again, but this got annoying.
 
So I opted to delete all the devices in the problematic location, which is when I noticed that dleleting some devices gave me errors when doing so.
 
errors like:
*Type: ObjectNotFound*
*Value: object  /zport/dmd/Devices/Server/Windows/Citrix/devices/SERVER32TS1 not found on  relation /zport/dmd/Monitors/Performance/localhost/devices*
Traceback (innermost last):
 
and:
*Type: ObjectNotFound*
*Value: object  /zport/dmd/Devices/Server/Windows/devices/SERVER54tsBROKEN/os/processes/ImaSrv.exe  not found on relation  /zport/dmd/Processes/Citrix/osProcessClasses/ImaSrv.exe/instances*
Traceback (innermost last):

 
or when i try to delete a process:
*Type: AttributeError*
*Value: 'tuple' object has no attribute 'index'*
Traceback (innermost last):

 
I'm still able to delete some devices, but clearly I'm hooped in some way. I was able to take a backup and restore to a test machine (which gives the same results). I thought maybe my catalog was the issue, I've read through all the forum discussions related to cataloging (zencatalog run --force) and layer3 fix (zendmd didn't work properly for me)... but not really sure where I should be looking anymore.
 
Any advice would be greatly appreciated.

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55291#55291





--end--

[yscotty]

just wanted to bump this once. I see similar threads with no responses either. crap. Back to the drawing board.

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55356#55356





--end--

[phonegi]

Did you try a zendmd *reindex()* and *commit()*?

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55448#55448





--end--

[yscotty]

I've always had problems with zendmd.
 
Am I right in the idea that I can't run zendmd as the zenoss user unless I stop all but thezeoctl services? That's the only way I've been able to run it.
I tried the reindex() and commit() ideas that you suggested, but I still get errors like this when deleting the problematic devices.
 
 
*Type: ObjectNotFound*
*Value: object <DeviceClass at Citrix> was not found on <ToOneRelationship at deviceClass>*
Traceback (innermost last):
 
 


--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55430#55430





--end--

[phonegi]

No you should be able to run zendmd from the command line as the zenoss user with the full system running. It requires a minimum of zeoctl and zopectl I believe, since it accesses the zope database.
 
Sounds like your zope database is pretty hosed. What errors do you get if you run zendmd?

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55433#55433





--end--

[yscotty]

~$ zendmd
2010-12-15 09:31:09 ERROR ZODB.lock_file Error locking file /usr/local/zenoss/zenoss/var/zeo1-1.zec.lock; pid=UNKNOWN
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/lib/python/ZODB/lock_file.py", line 80, in __init__
    _lock_file(fp)
  File "/usr/local/zenoss/zenoss/lib/python/ZODB/lock_file.py", line 59, in _lock_file
    raise LockError("Couldn't lock %r" % file.name)
LockError: Couldn't lock '/usr/local/zenoss/zenoss/var/zeo1-1.zec.lock'
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenModel/zendmd.py", line 245, in ?
    myconsole = HistoryConsole(locals=_customStuff())
  File "/usr/local/zenoss/zenoss/Products/ZenModel/zendmd.py", line 71, in _customStuff
    app = Zope2.app()
  File "/usr/local/zenoss/zenoss/lib/python/Zope2/__init__.py", line 51, in app
    startup()
  File "/usr/local/zenoss/zenoss/lib/python/Zope2/__init__.py", line 47, in startup
    _startup()
  File "/usr/local/zenoss/zenoss/lib/python/Zope2/App/startup.py", line 65, in startup
    DB = dbtab.getDatabase('/', is_root=1)
  File "/usr/local/zenoss/zenoss/lib/python/Zope2/Startup/datatypes.py", line 288, in getDatabase
    db = factory.open(name, self.databases)
  File "/usr/local/zenoss/zenoss/lib/python/Zope2/Startup/datatypes.py", line 186, in open
    DB = self.createDB(database_name, databases)
  File "/usr/local/zenoss/zenoss/lib/python/Zope2/Startup/datatypes.py", line 183, in createDB
    return ZODBDatabase.open(self, databases)
  File "/usr/local/zenoss/zenoss/lib/python/ZODB/config.py", line 97, in open
    storage = section.storage.open()
  File "/usr/local/zenoss/zenoss/lib/python/ZODB/config.py", line 168, in open
    realm=self.config.realm)
  File "/usr/local/zenoss/zenoss/lib/python/ZEO/ClientStorage.py", line 340, in __init__
    self._cache = self.ClientCacheClass(cache_path, size=cache_size)
  File "/usr/local/zenoss/zenoss/lib/python/ZEO/cache.py", line 186, in __init__
    self._lock_file = ZODB.lock_file.LockFile(path + '.lock')
  File "/usr/local/zenoss/zenoss/lib/python/ZODB/lock_file.py", line 80, in __init__
    _lock_file(fp)
  File "/usr/local/zenoss/zenoss/lib/python/ZODB/lock_file.py", line 59, in _lock_file
    raise LockError("Couldn't lock %r" % file.name)
ZODB.lock_file.LockError: Couldn't lock '/usr/local/zenoss/zenoss/var/zeo1-1.zec.lock'

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55476#55476





--end--

[phonegi]

What do you get with
ls -l /usr/local/zenoss/zenoss
and
ls -l /usr/local/zenoss/zenoss/var


--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55465#55465





--end--

[yscotty]

***@01zen1:~$ ls -l /usr/local/zenoss/zenoss
total 60
drwxr-x---  2 zenoss zenoss 4096 2010-09-29 12:45 backups
drwxr-xr-x  2 zenoss zenoss 4096 2010-12-07 17:07 bin
drwxr-xr-x  2 zenoss zenoss 4096 2010-09-02 10:22 doc
drwxr-xr-x  2 zenoss zenoss 4096 2010-12-14 16:40 etc
drwxr-xr-x  2 zenoss zenoss 4096 2010-09-02 10:22 Extensions
drwxr-xr-x  2 zenoss zenoss 4096 2010-09-02 10:22 import
drwxr-xr-x  3 zenoss zenoss 4096 2010-09-02 10:22 lib
drwxr-xr-x  2 zenoss zenoss 4096 2010-12-14 17:08 log
drwxr-xr-x  4 zenoss zenoss 4096 2009-08-25 16:46 perf
drwxr-xr-x 19 zenoss zenoss 4096 2010-10-12 14:19 Products
-rw-r--r--  1 zenoss zenoss  481 2010-09-02 10:22 README.txt
drwxr-xr-x  2 zenoss zenoss 4096 2010-09-02 10:22 scripts
drwxr-xr-x 10 zenoss zenoss 4096 2010-09-02 10:22 skel
drwxr-xr-x  2 zenoss zenoss 4096 2010-12-15 09:58 var
drwxr-x---  3 zenoss zenoss 4096 2010-12-07 17:09 ZenPacks
***@01zen1:~$ ls -l /usr/local/zenoss/zenoss/var
total 1102984
-rw-r--r-- 1 zenoss zenoss 367086154 2010-12-15 09:58 Data.fs
-rw-r--r-- 1 zenoss zenoss   7110019 2010-12-15 09:58 Data.fs.index
-rw-r--r-- 1 zenoss zenoss         6 2010-12-14 16:40 Data.fs.lock
-rw-r--r-- 1 zenoss zenoss 671525008 2010-12-15 09:30 Data.fs.old
-rw-r--r-- 1 zenoss zenoss    951417 2010-12-15 09:30 Data.fs.tmp
-rw-r--r-- 1 zenoss zenoss       299 2010-09-02 10:22 README.txt
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 Z2.lock
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 Z2.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenactions-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-07 15:18 zencatalog-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zencommand-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zeneventlog-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenhub-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenjobs-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenmodeler-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenperfsnmp-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenping-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenprocess-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenstatus-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zensyslog-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zentrap-localhost.pid
-rw-r--r-- 1 zenoss zenoss         4 2010-12-14 16:42 zenwin-localhost.pid
-rw-r--r-- 1 zenoss zenoss 209715200 2010-12-15 09:38 zeo1-1.zec
-rw-r--r-- 1 zenoss zenoss         0 2010-12-15 09:31 zeo1-1.zec.lock
-rw-r--r-- 1 zenoss zenoss         5 2010-12-14 16:40 ZEO.pid
srwx------ 1 zenoss zenoss         0 2010-12-14 16:42 zopectlsock

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55479#55479





--end--

[phonegi]

I think this thread (http://dev.zenoss.org/trac/ticket/5798) contains your solution. Seems there was a step in upgrading to your 2.5.2 system that may have been missed.
 
In the zope.conf file comment out the line:
     client zeo1
by changing it to:
     #client zeo1


--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55508#55508





--end--

[yscotty]

ah yes, that seems to make zendmd run with ease and without error.
 
however, I get errors such as this with reindex() and am still unable to remove certain devices or anything (including the device listed in the error)
 

Post by phonegi
reindex()

2010-12-15 15:41:17 WARNING zen.Device Unable to find 32TS1 (/zport/dmd/Devices/Server/Windows/Citrix/devices/32TS1) in organizer Groups/Citrix Servers
Traceback (most recent call last):
  File "<console>", line 1, in ?
  File "/usr/local/zenoss/zenoss/Products/ZenModel/zendmd.py", line 94, in reindex
    dmd.Devices.reIndex()
  File "/usr/local/zenoss/zenoss/Products/ZenModel/DeviceClass.py", line 774, in reIndex
    dev.index_object()
  File "/usr/local/zenoss/zenoss/Products/ZenModel/Device.py", line 1819, in index_object
    super(Device, self).index_object()
  File "/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.PreUpgrade30-1.1-py2.4.egg/ZenPacks/zenoss/PreUpgrade30/__init__.py", line 37, in new_index_object
    old_index_object(self)
  File "/usr/local/zenoss/zenoss/Products/ZenModel/ZenModelRM.py", line 354, in index_object
    cat.catalog_object(self, self.getPrimaryId())
  File "/usr/local/zenoss/zenoss/lib/python/Products/ZCatalog/ZCatalog.py", line 536, in catalog_object
    update_metadata=update_metadata)
  File "/usr/local/zenoss/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 339, in catalogObject
    index = self.updateMetadata(object, uid)
  File "/usr/local/zenoss/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 277, in updateMetadata
    newDataRecord = self.recordify(object)
  File "/usr/local/zenoss/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 417, in recordify
    if(attr is not MV and safe_callable(attr)): attr=attr()
  File "/usr/local/zenoss/zenoss/Products/ZenModel/Device.py", line 550, in path
    rel = org.primaryAq().devices
  File "/usr/local/zenoss/zenoss/Products/ZenRelations/PrimaryPathObjectManager.py", line 80, in primaryAq
    return getObjByPath(app, self.getPrimaryPath())
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Utils.py", line 239, in getObjByPath
    next = bobo_traverse(REQUEST, name)
  File "/usr/local/zenoss/zenoss/lib/python/OFS/Application.py", line 135, in __bobo_traverse__
    raise KeyError, name
KeyError: 'Lloydminster'
 
 
yikes!

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55489#55489





--end--

[phonegi]

When an object is deleted, its relationships have to be removed. What you have is a bunch of broken relationships. Each device has a relationship named *perfServer* that points to the performance monitor object (class *PerformanceConf*) that is responsible for collecting performance data. Likewise the performance monitor object has a relationship named *devices* that points to a list of device objects. It looks like your devices have *perfServer* relationships pointing to the performance monitor, but the performance monitor doesn't contain the second half of the relationship.
 
You need to execute code that will find the broken relationships and mend them.This will modify your zope db at a very low level. I suggest you perform a backup before executing this code.
 
Enter zendmd. Paste this code:
 
def fixRelationships():
    for device in dmd.Devices.getSubDevices():
        if device.perfServer():
            perfServer = device.perfServer()
            if not perfServer.devices.hasobject( device ):
                perfServer.devices._add( device )
 
Then enter:
 

fixRelationships()
reindex()
commit()

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55514#55514





--end--

[yscotty]

Ok, I ran the script. I still get the exact same previous errors on the reindex() command.
I seem to be able to delete some devices (still not others, but at least more now)..... but... now i'm not able to search for devices.. and monitoring seems to be broken to everything else. By that I mean.. the current open events aren't being monitored.
And upon further inspection... since I ran the script, graphs aren't updating, etc.
 
I did take a backup, so I can revert. I'm going to try running that script on an older backup (when the problem first started) and go from there.

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55535#55535





--end--

[phonegi]

zenperfsnmp probably needs to be restarted.

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55522#55522





--end--

[yscotty]

no, I've pretty much restarted everything a number of times. Nothing, except for devices I just added, are being monitored any more.

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55536#55536





--end--

[yscotty]

zenrestored my way back to a working zenoss from this morning.

--------------------------------------------------------------

To post a reply to the thread message, either reply to this email (recommended) or visit the message page: http://community.zenoss.org/message/55537#55537





--end--