[ZC4.2SP1] zenjobs not running: celery.worker: permission denied

[Pim Rupert]

Pim Rupert [http://community.zenoss.org/people/prupert] created the discussion

"[ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68451#68451

--------------------------------------------------------------
Hi all,

We are having a problem on a new (clean) installation of Zenoss Core 4.2 with Service Pack 1 applied on a new (clean) machine with CentOS 6.

The daemon zenjobs is not running.

---- **** -----
--- * ***  * -- [Configuration]
- ** ----------   . loader:      Products.ZenUtils.celeryintegration.loader.ZenossLoader
- ** ----------   . concurrency: 2
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
--------------   . celery:      exchange:celery (direct) binding:celery
[Tasks]
  . Products.Jobber.facade.FacadeMethodJob
  . Products.Jobber.jobs.SubprocessJob
  . Products.Jobber.zenmodel.DeviceSetLocalRolesJob
  . Products.ZenModel.IpNetwork.AutoDiscoveryJob
  . Products.ZenModel.ZDeviceLoader.DeviceCreationJob
2012-09-10 13:25:45,110 ERROR celery.worker: Unrecoverable error: OSError(13, 'Permission denied')
  File "/opt/zenoss/lib/python/celery/worker/__init__.py", line 238, in start
    component.start()
  File "/opt/zenoss/lib/python/celery/concurrency/base.py", line 85, in start
    self.on_start()
  File "/opt/zenoss/lib/python/celery/concurrency/processes/__init__.py", line 67, in on_start
    **self.options)
  File "/opt/zenoss/lib/python/billiard/pool.py", line 575, in __init__
    self._setup_queues()
  File "/opt/zenoss/lib/python/billiard/pool.py", line 754, in _setup_queues
    self._inqueue = SimpleQueue()
  File "/opt/zenoss/lib/python/billiard/queues.py", line 340, in __init__
    self._rlock = Lock()
  File "/opt/zenoss/lib/python/billiard/synchronize.py", line 171, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/opt/zenoss/lib/python/billiard/synchronize.py", line 85, in __init__
    kind, value, maxvalue, self._make_name(), unlink_immediately)
OSError: [Errno 13] Permission denied
2012-09-10 13:25:45,114 INFO multiprocessing: process shutting down

Any help would be greatly appreciated. Thanks.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68451#68451]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68477#68477

--------------------------------------------------------------

rabbitmqctl list_user_permissions -p zenoss zenoss

and provide the output?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68477#68477]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Pim Rupert]

Pim Rupert [http://community.zenoss.org/people/prupert] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68478#68478

--------------------------------------------------------------

# rabbitmqctl list_user_permissions -p zenoss zenoss
Listing permissions for user "zenoss" ...
/zenoss          .*          .*          .*
...done.

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68478#68478]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68495#68495

--------------------------------------------------------------
those appear to be the proper permissions. Let see if we can get more output. Try running (as the zenoss user)

zenjobs run -v 10

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68495#68495]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Pim Rupert]

Pim Rupert [http://community.zenoss.org/people/prupert] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68483#68483

--------------------------------------------------------------

# su - zenoss
$ zenjobs run -v 10
2012-09-11 15:09:42,287 INFO celery.redirected: 
---- **** -----
--- * ***  * -- [Configuration]
- ** ----------   . loader:      Products.ZenUtils.celeryintegration.loader.ZenossLoader
- ** ----------   . concurrency: 2
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
--------------   . celery:      exchange:celery (direct) binding:celery
[Tasks]
  . Products.Jobber.facade.FacadeMethodJob
  . Products.Jobber.jobs.SubprocessJob
  . Products.Jobber.zenmodel.DeviceSetLocalRolesJob
  . Products.ZenModel.IpNetwork.AutoDiscoveryJob
  . Products.ZenModel.ZDeviceLoader.DeviceCreationJob
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.contrib.abortable.AbortableTask
  . celery.group
2012-09-11 15:09:42,288 DEBUG celery.worker.abstract: [Worker] Loading modules.
2012-09-11 15:09:42,293 DEBUG celery.worker.abstract: [Worker] Claiming components.
2012-09-11 15:09:42,293 DEBUG celery.worker.abstract: [Worker] Building boot step graph.
2012-09-11 15:09:42,293 DEBUG celery.worker.abstract: [Worker] New boot order: ['queues', 'pool', 'mediator', 'beat', 'autoreloader', 'timers', 'state-db', 'autoscaler', 'consumer']
2012-09-11 15:09:42,297 DEBUG celery.worker: Starting celery.concurrency.processes.TaskPool...
2012-09-11 15:09:42,299 ERROR celery.worker: Unrecoverable error: OSError(13, 'Permission denied')
  File "/opt/zenoss/lib/python/celery/worker/__init__.py", line 238, in start
    component.start()
  File "/opt/zenoss/lib/python/celery/concurrency/base.py", line 85, in start
    self.on_start()
  File "/opt/zenoss/lib/python/celery/concurrency/processes/__init__.py", line 67, in on_start
    **self.options)
  File "/opt/zenoss/lib/python/billiard/pool.py", line 575, in __init__
    self._setup_queues()
  File "/opt/zenoss/lib/python/billiard/pool.py", line 754, in _setup_queues
    self._inqueue = SimpleQueue()
  File "/opt/zenoss/lib/python/billiard/queues.py", line 340, in __init__
    self._rlock = Lock()
  File "/opt/zenoss/lib/python/billiard/synchronize.py", line 171, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/opt/zenoss/lib/python/billiard/synchronize.py", line 85, in __init__
    kind, value, maxvalue, self._make_name(), unlink_immediately)
OSError: [Errno 13] Permission denied
2012-09-11 15:09:42,302 INFO multiprocessing: process shutting down
2012-09-11 15:09:42,302 DEBUG multiprocessing: running all "atexit" finalizers with priority >= 0
2012-09-11 15:09:42,302 DEBUG multiprocessing: running the remaining "atexit" finalizers

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68483#68483]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Pim Rupert]

Pim Rupert [http://community.zenoss.org/people/prupert] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68540#68540

--------------------------------------------------------------
Does anybody have an idea how to tackle this issue?

We have used a bare CentOS 6 machine and the official installation script. We really would like to start testing with Zenoss 4.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68540#68540]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jplouis]

jplouis [http://community.zenoss.org/people/jplouis] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68542#68542

--------------------------------------------------------------
Check your permissions on /dev/shm. The zenoss user should have rwx on it, should look something like:

drwxrwxrwt.  2 root root          40 Sep 12 11:20 shm
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68542#68542]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Pim Rupert]

Pim Rupert [http://community.zenoss.org/people/prupert] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68543#68543

--------------------------------------------------------------

Post by jplouis
drwxrwxrwt.  2 root root          40 Sep 12 11:20 shm

/dev/shm was indeed not writable. I have changed permissions on /dev/shm to 1777 and am now able to start up the zenjobs daemon without trouble. Thank you.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68543#68543]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Pim Rupert]

Pim Rupert [http://community.zenoss.org/people/prupert] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68713#68713

--------------------------------------------------------------
It is 0755 by default on a fresh CentOS 6 box. Shouldn't the Zenoss installer take care of fixing (or at least, checking) these permissions?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68713#68713]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68716#68716

--------------------------------------------------------------
This seems like a very reasonable expectation. I'd file file a bug at http://jira.zenoss.com/ http://jira.zenoss.com/
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68716#68716]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Pim Rupert]

Pim Rupert [http://community.zenoss.org/people/prupert] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/68717#68717

--------------------------------------------------------------

Post by dpetzel
This seems like a very reasonable expectation. I'd file file a bug at http://jira.zenoss.com/ http://jira.zenoss.com/

I already have. :)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68717#68717]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[PaVan Jakati]

PaVan Jakati [http://community.zenoss.org/people/PaVan] created the discussion

"Re: [ZC4.2SP1] zenjobs not running: celery.worker: permission denied"

To view the discussion, visit: http://community.zenoss.org/message/73608#73608

--------------------------------------------------------------
I am getting simillar error when try on run zenjobs on fresh installtion . CentOs 6.3


2013-06-12 21:20:00,266 INFO celery.redirected:


-------------- ***@RMBRZENOSS v2.6.0a3
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      amqp://***@localhost:5672//zenoss
- ** ----------   . loader:      Products.ZenUtils.celeryintegration.loader.ZenossLoader
- ** ----------   . logfile:     [stderr]@INFO
- ** ----------   . concurrency: 2
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
--------------   . celery:      exchange:celery (direct) binding:celery




[Tasks]
  . Products.Jobber.facade.FacadeMethodJob
  . Products.Jobber.jobs.SubprocessJob
  . Products.Jobber.zenmodel.DeviceSetLocalRolesJob
  . Products.ZenModel.IpNetwork.AutoDiscoveryJob
  . Products.ZenModel.ZDeviceLoader.DeviceCreationJob


2013-06-12 21:20:00,275 INFO multiprocessing: child process calling self.run()
2013-06-12 21:20:00,278 INFO multiprocessing: child process calling self.run()
2013-06-12 21:20:00,279 INFO celery.redirected: ***@RMBRZENOSS has started.
^C2013-06-12 21:20:20,048 INFO celery.redirected: celeryd: Hitting Ctrl+C again will terminate all running tasks!
2013-06-12 21:20:20,048 INFO celery.redirected: celeryd: Warm shutdown (MainProcess)
^C2013-06-12 21:20:20,289 INFO celery.redirected: celeryd: Cold shutdown (MainProcess)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73608#73608]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]