Discussion:
/tmp filling up - process still holding deleted files
Craig Ward
2013-05-29 22:25:03 UTC
Permalink
Craig Ward [http://community.zenoss.org/people/welshwardy] created the discussion

"/tmp filling up - process still holding deleted files"

To view the discussion, visit: http://community.zenoss.org/message/73355#73355

--------------------------------------------------------------
Hi,

Our /tmp directory is being reported as full:

***@zenoss:/tmp# df -h | grep /tmp
                      535M  499M   31M  95% /tmp

Yet the contents are pretty small:

***@zenoss:/tmp# du -sh /tmp
179K    /tmp


Looks like a Zenoss process is holding open lots of deleted files:

***@zenoss:/tmp# lsof /tmp | more
COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
runzope   19733 zenoss    7u   REG  254,4    91272   11 /tmp/tmpWk20uM.tbuf (deleted)
runzope   19733 zenoss   11u   REG  254,4 20971520   12 /tmp/tmphMTSsI (deleted)
.python.b 19777 zenoss    4u   REG  254,4   170911   13 /tmp/tmpzwz9C8.tbuf (deleted)
.python.b 19777 zenoss    5u   REG  254,4 10485760   14 /tmp/tmpsiul16 (deleted)
.python.b 19814 zenoss    6u   REG  254,4        0   15 /tmp/tmpphUJhF.tbuf (deleted)
.python.b 19814 zenoss    7u   REG  254,4 10485760   16 /tmp/tmpYzgKfo (deleted)
.python.b 19966 zenoss    4u   REG  254,4     5510   17 /tmp/tmp3IZBEh.tbuf (deleted)
.python.b 19966 zenoss    7u   REG  254,4 10485760   18 /tmp/tmpIxJmIf (deleted)
.python.b 20117 zenoss    7u   REG  254,4     1835   19 /tmp/tmpO8WxHP.tbuf (deleted)
.python.b 20117 zenoss    8u   REG  254,4 10485760   20 /tmp/tmpFyf6Om (deleted)
...

***@zenoss:/tmp# ps auxww | grep 20117
root     18330  0.0  0.0   7548   848 pts/0    S+   10:23   0:00 grep 20117
zenoss   20117 20.6  3.0 347044 123452 ?       Sl   May29 298:04 /usr/local/zenoss/python/bin/.python.bin /usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py --configfile /usr/local/zenoss/zenoss/etc/zencommand.conf --cycle --daemon


Any ideas what could be causing the files to be deleted but still held open?


Thanks!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73355#73355]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
mwcotton
2013-06-03 00:06:48 UTC
Permalink
mwcotton [http://community.zenoss.org/people/mwcotton] created the discussion

"Re: /tmp filling up - process still holding deleted files"

To view the discussion, visit: http://community.zenoss.org/message/73442#73442

--------------------------------------------------------------
I found they only way for them to go away is to restart zenoss.

+Any ideas what could be causing the files to be deleted but still held open?+
  this actually happens often on a linux/unix box its something like the files are deleted in the "dir" or listing but the space is not returned to the filesystem until the process that is holding them open releases them.

There is supposedly some configuration file where you can adjust where these temp files are placed but I could not find it.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73442#73442]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Mike Bordignon
2013-07-09 22:04:57 UTC
Permalink
Mike Bordignon [http://community.zenoss.org/people/mab] created the discussion

"Re: /tmp filling up - process still holding deleted files"

To view the discussion, visit: http://community.zenoss.org/message/73891#73891

--------------------------------------------------------------
I'm also having this problem - certain Zenoss processes (mainly zencommand) appear to delete files but not release the inode afterwards, hence the filesystem appears to be 'full' of deleted files.

# lsof | grep -c 'deleted'
661

# lsof | grep 'deleted'
.python.b  8895      zenoss    7u      REG              254,4      1835         21 /tmp/tmpXizW_h.tbuf (deleted)
.python.b  8895      zenoss    8u      REG              254,4  10485760         22 /tmp/tmp1bk3ck (deleted)
.python.b  8895      zenoss   19u      REG              254,4      1835         23 /tmp/tmptkAphL.tbuf (deleted)
.python.b  8895      zenoss   20u      REG              254,4  10485760         24 /tmp/tmp8YH_gD (deleted)
.python.b  8895      zenoss   22u      REG              254,4      1796         25 /tmp/tmpRWyEZz.tbuf (deleted)
.python.b  8895      zenoss   23u      REG              254,4  10485760         26 /tmp/tmpDTDPe4 (deleted)
.python.b  8895      zenoss   25u      REG              254,4      1835         27 /tmp/tmpGZXGQs.tbuf (deleted)
.python.b  8895      zenoss   26u      REG              254,4  10485760         28 /tmp/tmpYTxZR0 (deleted)
.python.b  8895      zenoss   28u      REG              254,4      1796         29 /tmp/tmpiMhCrj.tbuf (deleted)
.python.b  8895      zenoss   29u      REG              254,4  10485760         30 /tmp/tmpPp08xk (deleted)
.python.b  8895      zenoss   31u      REG              254,4      1835         31 /tmp/tmp9eaOs5.tbuf (deleted)
.python.b  8895      zenoss   32u      REG              254,4  10485760         32 /tmp/tmpXC0IDr (deleted)
.python.b  8895      zenoss   34u      REG              254,4      1796         33 /tmp/tmpWQRnEs.tbuf (deleted)
.python.b  8895      zenoss   35u      REG              254,4  10485760         34 /tmp/tmpYLI0C0 (deleted)
.python.b  8895      zenoss   37u      REG              254,4      1835         35 /tmp/tmpaakyyw.tbuf (deleted)
.python.b  8895      zenoss   38u      REG              254,4  10485760         36 /tmp/tmpiMqfhX (deleted)
.python.b  8895      zenoss   40u      REG              254,4      1796         37 /tmp/tmpY_91m9.tbuf (deleted)
.python.b  8895      zenoss   41u      REG              254,4  10485760         38 /tmp/tmpeFXRlg (deleted)
.python.b  8895      zenoss   43u      REG              254,4      1835         39 /tmp/tmpcBo6Mz.tbuf (deleted)
.python.b  8895      zenoss   44u      REG              254,4  10485760         40 /tmp/tmp0vrFiG (deleted)
.python.b  8895      zenoss   46u      REG              254,4      1796         41 /tmp/tmpkY5nIN.tbuf (deleted)
.python.b  8895      zenoss   47u      REG              254,4  10485760         42 /tmp/tmpVGIDNH (deleted)
.python.b  8895      zenoss   49u      REG              254,4      1835         43 /tmp/tmpTiTL4t.tbuf (deleted)
.python.b  8895      zenoss   50u      REG              254,4  10485760         44 /tmp/tmppf2MXT (deleted)
.python.b  8895      zenoss   52u      REG              254,4      1796         45 /tmp/tmpagO7MD.tbuf (deleted)
.python.b  8895      zenoss   53u      REG              254,4  10485760         46 /tmp/tmpWsw5_k (deleted)
.python.b  8895      zenoss   55u      REG              254,4      1835         47 /tmp/tmpsdua5O.tbuf (deleted)
.python.b  8895      zenoss   56u      REG              254,4  10485760         48 /tmp/tmptN3vdp (deleted)
.python.b  8895      zenoss   58u      REG              254,4      1796         49 /tmp/tmpHOnlL6.tbuf (deleted)
.python.b  8895      zenoss   59u      REG              254,4  10485760         50 /tmp/tmpL21XJI (deleted)
.python.b  8895      zenoss   61u      REG              254,4      1835         51 /tmp/tmpGGd9dD.tbuf (deleted)
.python.b  8895      zenoss   62u      REG              254,4  10485760         52 /tmp/tmprgiFeP (deleted)
.python.b  8895      zenoss   64u      REG              254,4      1796         53 /tmp/tmpJ8Atzz.tbuf (deleted)
.python.b  8895      zenoss   65u      REG              254,4  10485760         54 /tmp/tmpCR2s30 (deleted)
.python.b  8895      zenoss   67u      REG              254,4      1835         55 /tmp/tmpd_ne6r.tbuf (deleted)
.python.b  8895      zenoss   68u      REG              254,4  10485760         56 /tmp/tmp1lVKWl (deleted)
.python.b  8895      zenoss   70u      REG              254,4      1796         57 /tmp/tmpMoaBzj.tbuf (deleted)
.python.b  8895      zenoss   71u      REG              254,4  10485760         58 /tmp/tmpSuPqHt (deleted)
.python.b  8895      zenoss   73u      REG              254,4      1835         59 /tmp/tmpJV5EQV.tbuf (deleted)
.python.b  8895      zenoss   74u      REG              254,4  10485760         60 /tmp/tmp8aveCa (deleted)
.python.b  8895      zenoss   76u      REG              254,4      1796         61 /tmp/tmpKKtwZI.tbuf (deleted)
.python.b  8895      zenoss   77u      REG              254,4  10485760         62 /tmp/tmpRSiwvA (deleted)
.python.b  8895      zenoss   79u      REG              254,4      1835         63 /tmp/tmpbVeq3B.tbuf (deleted)
.python.b  8895      zenoss   80u      REG              254,4  10485760         64 /tmp/tmp2wxNw9 (deleted)
.python.b  8895      zenoss   82u      REG              254,4      1796         65 /tmp/tmpvB6etK.tbuf (deleted)
.python.b  8895      zenoss   83u      REG              254,4  10485760         66 /tmp/tmpxqZjE1 (deleted)
.python.b  8895      zenoss   85u      REG              254,4      1835         67 /tmp/tmpZKMOxf.tbuf (deleted)
.python.b  8895      zenoss   86u      REG              254,4  10485760         68 /tmp/tmpkoBLnY (deleted)
.python.b  8895      zenoss   88u      REG              254,4      1796         69 /tmp/tmpMaDspN.tbuf (deleted)
.python.b  8895      zenoss   89u      REG              254,4  10485760         70 /tmp/tmpISNFTw (deleted)
.python.b  8895      zenoss   91u      REG              254,4      1835         71 /tmp/tmpyiV1AN.tbuf (deleted)
.python.b  8895      zenoss   92u      REG              254,4  10485760         72 /tmp/tmprW_X9z (deleted)


This is a massive problem for me - surely others have had the same issue? Help me Zenoss Gods!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73891#73891]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Vaibhav.Narula
2013-07-10 18:47:10 UTC
Permalink
Vaibhav.Narula [http://community.zenoss.org/people/Vaibhav.Narula] created the discussion

"Re: /tmp filling up - process still holding deleted files"

To view the discussion, visit: http://community.zenoss.org/message/73918#73918

--------------------------------------------------------------
which version of zenoss are  you using and  are  you using fping zenpack  by any chance  as  i saw some issue  with it in the past
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73918#73918]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Loading...