Discussion:
Zenoss 4.2 RAM Utilization/Other Issues
Justin Simmons
2013-01-08 21:40:15 UTC
Permalink
Justin Simmons [http://community.zenoss.org/people/jmsimmons] created the discussion

"Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70837#70837

--------------------------------------------------------------
Recently we upgraded from 3.2 to 4.2. Since the upgrade, we have been experiencing memory utilization that I have to believe is due to a memory leak. I have to reboot our Zenoss server daily, and have to restart the zentrap and zenperfsnmp services about every 4 hours.

I'm also noticing a lot of events (I'm attributing to my memory issues) popping up saying:

"Scan stopped; Collection time exceeded interval - Elapsed time +########+ seconds greater than 300 seconds.

Our server is monitoring 700 devices, running CentOS 5.7, 4 CPUs and 16GB of RAM.

I'm somewhat a newbie to Zenoss, and I'm a complete newbie to python scripting, etc., so any help on solving my memory usage issue would be helpful.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70837#70837]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
hydruid
2013-01-09 00:07:04 UTC
Permalink
hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70882#70882

--------------------------------------------------------------
Justin,

Take a look at this: http://wiki.zenoss.org/Zenoss_Core_4.2.0_SP1 http://wiki.zenoss.org/Zenoss_Core_4.2.0_SP1

Install SP1 if you haven't already, it is supposed to fix a memory leak issue with PySamba

Hydruid
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70882#70882]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Justin Simmons
2013-01-10 14:59:06 UTC
Permalink
Justin Simmons [http://community.zenoss.org/people/jmsimmons] created the discussion

"Re: Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70911#70911

--------------------------------------------------------------
I went ahead and installed the SP...will provide an update on performance tomorrow.

Also, after installing the SP, I'm still seeing these errors showing up saying:

"Scan stopped; Collection time exceeded interval - Elapsed time #### seconds greater than 300 seconds"

This error seems to be why I'm seeing intermittent gaps in my graph data.

I haven't seen any other posts with this error, so if anyone has a solution, please assist.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70911#70911]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Justin Simmons
2013-01-10 16:24:29 UTC
Permalink
Justin Simmons [http://community.zenoss.org/people/jmsimmons] created the discussion

"Re: Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70917#70917

--------------------------------------------------------------
Well no need to wait. Out of 16GB of RAM, all but about 80MB are being chewed up by Zenoss. Is this normal?

I've posted a ps aux output of the running processes. Note that zentrap.py is utilizing about 6GB of this. Is this normal as well?

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  10364   696 ?        Ss   08:42   0:00 init [3]
root         2  0.0  0.0      0     0 ?        S<   08:42   0:01 [migration/0]
root         3  1.4  0.0      0     0 ?        SN   08:42   2:15 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   08:42   0:00 [migration/1]
root         5  0.0  0.0      0     0 ?        SN   08:42   0:00 [ksoftirqd/1]
root         6  0.0  0.0      0     0 ?        S<   08:42   0:00 [migration/2]
root         7  0.0  0.0      0     0 ?        SN   08:42   0:00 [ksoftirqd/2]
root         8  0.0  0.0      0     0 ?        S<   08:42   0:00 [migration/3]
root         9  0.0  0.0      0     0 ?        SN   08:42   0:00 [ksoftirqd/3]
root        10  0.4  0.0      0     0 ?        S<   08:42   0:42 [events/0]
root        11  0.0  0.0      0     0 ?        S<   08:42   0:00 [events/1]
root        12  0.0  0.0      0     0 ?        S<   08:42   0:00 [events/2]
root        13  0.0  0.0      0     0 ?        S<   08:42   0:00 [events/3]
root        14  0.0  0.0      0     0 ?        S<   08:42   0:00 [khelper]
root       151  0.0  0.0      0     0 ?        S<   08:42   0:00 [kthread]
root       158  0.6  0.0      0     0 ?        S<   08:42   0:56 [kblockd/0]
root       159  0.0  0.0      0     0 ?        S<   08:42   0:00 [kblockd/1]
root       160  0.0  0.0      0     0 ?        S<   08:42   0:00 [kblockd/2]
root       161  0.0  0.0      0     0 ?        S<   08:42   0:00 [kblockd/3]
root       162  0.0  0.0      0     0 ?        S<   08:42   0:00 [kacpid]
root       323  0.0  0.0      0     0 ?        S<   08:42   0:00 [cqueue/0]
root       324  0.0  0.0      0     0 ?        S<   08:42   0:00 [cqueue/1]
root       325  0.0  0.0      0     0 ?        S<   08:42   0:00 [cqueue/2]
root       326  0.0  0.0      0     0 ?        S<   08:42   0:00 [cqueue/3]
root       329  0.0  0.0      0     0 ?        S<   08:42   0:00 [khubd]
root       331  0.0  0.0      0     0 ?        S<   08:42   0:00 [kseriod]
root       423  0.0  0.0      0     0 ?        S    08:42   0:00 [khungtaskd]
root       424  0.0  0.0      0     0 ?        S    08:42   0:00 [pdflush]
root       425  2.4  0.0      0     0 ?        S    08:42   3:44 [pdflush]
root       426  0.0  0.0      0     0 ?        S<   08:42   0:00 [kswapd0]
root       427  0.0  0.0      0     0 ?        S<   08:42   0:00 [aio/0]
root       428  0.0  0.0      0     0 ?        S<   08:42   0:00 [aio/1]
root       429  0.0  0.0      0     0 ?        S<   08:42   0:00 [aio/2]
root       430  0.0  0.0      0     0 ?        S<   08:42   0:00 [aio/3]
root       636  0.0  0.0      0     0 ?        S<   08:42   0:00 [kpsmoused]
root       699  0.0  0.0      0     0 ?        S<   08:42   0:00 [mpt_poll_0]
root       700  0.0  0.0      0     0 ?        S<   08:42   0:00 [mpt/0]
root       701  0.0  0.0      0     0 ?        S<   08:42   0:00 [scsi_eh_0]
root       707  0.0  0.0      0     0 ?        S<   08:42   0:00 [ata/0]
root       708  0.0  0.0      0     0 ?        S<   08:42   0:00 [ata/1]
root       709  0.0  0.0      0     0 ?        S<   08:42   0:00 [ata/2]
root       710  0.0  0.0      0     0 ?        S<   08:42   0:00 [ata/3]
root       711  0.0  0.0      0     0 ?        S<   08:42   0:00 [ata_aux]
root       727  0.0  0.0      0     0 ?        S<   08:42   0:00 [kstriped]
root       748  0.5  0.0      0     0 ?        S<   08:43   0:52 [kjournald]
root       773  0.0  0.0      0     0 ?        S<   08:43   0:00 [kauditd]
root       806  0.0  0.0  12708   868 ?        S<s  08:43   0:00 /sbin/udevd -d
root      2097  0.0  0.0      0     0 ?        S<   08:43   0:00 [kmpathd/0]
root      2098  0.0  0.0      0     0 ?        S<   08:43   0:00 [kmpathd/1]
root      2099  0.0  0.0      0     0 ?        S<   08:43   0:00 [kmpathd/2]
root      2100  0.0  0.0      0     0 ?        S<   08:43   0:00 [kmpathd/3]
root      2101  0.0  0.0      0     0 ?        S<   08:43   0:00 [kmpath_handlerd]
root      2129  0.0  0.0      0     0 ?        S<   08:43   0:00 [kjournald]
root      2425  0.0  0.0      0     0 ?        S<   08:43   0:00 [vmmemctl]
root      2606  0.0  0.0  46640  2840 ?        S    08:43   0:01 /usr/sbin/vmtoolsd
root      2649  0.0  0.0      0     0 ?        S<   08:43   0:00 [iscsi_eh]
root      2728  0.0  0.0      0     0 ?        S<   08:43   0:00 [cnic_wq]
root      2751  0.0  0.0      0     0 ?        S<   08:43   0:00 [ib_addr]
root      2768  0.0  0.0      0     0 ?        S<   08:43   0:00 [ib_mcast]
root      2769  0.0  0.0      0     0 ?        S<   08:43   0:00 [ib_inform]
root      2770  0.0  0.0      0     0 ?        S<   08:43   0:00 [local_sa]
root      2776  0.0  0.0      0     0 ?        S<   08:43   0:00 [iw_cm_wq]
root      2783  0.0  0.0      0     0 ?        S<   08:43   0:00 [ib_cm/0]
root      2784  0.0  0.0      0     0 ?        S<   08:43   0:00 [ib_cm/1]
root      2785  0.0  0.0      0     0 ?        S<   08:43   0:00 [ib_cm/2]
root      2786  0.0  0.0      0     0 ?        S<   08:43   0:00 [ib_cm/3]
root      2795  0.0  0.0      0     0 ?        S<   08:43   0:00 [rdma_cm]
root      2814  0.0  0.0  38876   448 ?        Ssl  08:43   0:00 brcm_iscsiuio
root      2822  0.0  0.0  12148   464 ?        Ss   08:43   0:00 iscsid
root      2823  0.0  0.0  12644  4460 ?        S<Ls 08:43   0:00 iscsid
root      3023  0.0  0.0   5924   624 ?        Ss   08:43   0:00 syslogd -m 0
root      3026  0.0  0.0   3820   436 ?        Ss   08:43   0:00 klogd -x
root      3055  0.0  0.0 161612  9068 ?        Sl   08:43   0:05 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
104       3071  0.7  1.2 279656 205336 ?       Ssl  08:43   1:09 memcached -d -p 11211 -u memcached -m 1024 -c 1024 -P /var/run/memcached/me
root      3092  0.0  0.0  62652  1216 ?        Ss   08:43   0:00 /usr/sbin/sshd
ntp       3109  0.0  0.0  19208  4904 ?        SLs  08:43   0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      3118  0.0  0.0  65944  1344 ?        S    08:43   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/m
mysql     3232  1.6  5.6 1578568 929896 ?      Sl   08:43   2:32 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/
rabbitmq  3292  0.0  0.0  10512   288 ?        S    08:43   0:00 /usr/lib64/erlang/erts-5.6.5/bin/epmd -daemon
root      3308  0.0  0.0  90980  1248 ?        S    08:43   0:00 runuser rabbitmq --session-command /usr/sbin/rabbitmq-server
rabbitmq  3330  0.0  0.0  63848  1092 ?        S    08:43   0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq  3344  0.7  0.4 157800 66776 ?        Sl   08:43   1:08 /usr/lib64/erlang/erts-5.6.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -
rabbitmq  3432  0.0  0.0  10512   432 ?        Ss   08:43   0:00 inet_gethost 4
rabbitmq  3433  0.0  0.0  12604   536 ?        S    08:43   0:00 inet_gethost 4
root      3457  0.0  0.0  66964  2084 ?        Ss   08:43   0:00 sendmail: accepting connections
smmsp     3465  0.0  0.0  57716  1780 ?        Ss   08:43   0:00 sendmail: Queue ***@01:00:00 for /var/spool/clientmqueue
zenoss    4165  2.0  2.1 1167220 355544 ?      Sl   08:44   3:11 java -server -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -DZENOSS_COMMAND=zene
zenoss    4295  0.0  0.0 156684  7404 ?        Ssl  08:44   0:00 /opt/zenoss/bin/python /opt/zenoss/lib/python/zdaemon/zdrun.py -S /opt/zeno
zenoss    4297  7.2  2.7 875972 454408 ?       Sl   08:44  10:59 /opt/zenoss/bin/python /opt/zenoss/zopehome/runzope -C /opt/zenoss/etc/zope
zenoss    4345 37.2  3.7 969168 617880 ?       S    08:44  56:28 /opt/zenoss/bin/python /opt/zenoss/Products/ZenHub/zenhub.py --configfile /
zenoss    4408  0.0  0.5 483584 95196 ?        Sl   08:44   0:02 /opt/zenoss/bin/python /opt/zenoss/Products/Jobber/zenjobs.py --configfile
zenoss    4425  0.0  0.6 485344 108900 ?       S    08:44   0:01 /opt/zenoss/bin/python /opt/zenoss/Products/Jobber/zenjobs.py --configfile
zenoss    4426  0.0  0.5 442608 94028 ?        S    08:44   0:00 /opt/zenoss/bin/python /opt/zenoss/Products/Jobber/zenjobs.py --configfile
zenoss    4458 15.8  1.0 519544 170156 ?       S    08:44  23:54 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zeneventd.py --config
zenoss    4503  0.8  0.6 450560 109892 ?       S    08:44   1:16 /opt/zenoss/bin/python /opt/zenoss/Products/ZenStatus/zenping.py --configfi
zenoss    4564  0.0  0.5 435392 94180 ?        S    08:44   0:08 /opt/zenoss/bin/python /opt/zenoss/Products/ZenStatus/zenstatus.py --config
zenoss    4605  0.0  0.5 449796 98256 ?        S    08:45   0:05 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zenactiond.py --confi
zenoss    4741  1.8  1.4 602968 233572 ?       S    08:45   2:44 /opt/zenoss/bin/python /opt/zenoss/Products/DataCollector/zenmodeler.py --c
zenoss    4758  0.0  0.5 436564 95376 ?        S    08:45   0:01 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zensyslog.py --config
zenoss    4783 18.1  3.6 948072 607912 ?       S    08:45  27:16 /opt/zenoss/bin/python /opt/zenoss/Products/ZenRRD/zenperfsnmp.py --configf
zenoss    4813  0.0  0.5 435552 94324 ?        S    08:45   0:01 /opt/zenoss/bin/python /opt/zenoss/Products/ZenRRD/zencommand.py --configfi
zenoss    4844  0.1  0.5 435628 95260 ?        S    08:45   0:09 /opt/zenoss/bin/python /opt/zenoss/Products/ZenRRD/zenprocess.py --configfi
zenoss    4849  5.8  0.3 405652 61320 ?        Ssl  08:45   8:44 /usr/bin/rrdcached -b /opt/zenoss/perf -p /opt/zenoss/var/rrdcached.pid -l
zenoss    4891  0.0  0.5 435504 94280 ?        S    08:45   0:01 /opt/zenoss/bin/python /opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.8.0-py
zenoss    4898  0.0  0.1 915244 30588 ?        Sl   08:45   0:01 java -server -Xmx512m -jar /opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.8.
zenoss    4964  0.0  0.5 462320 94896 ?        S    08:45   0:01 /opt/zenoss/bin/python /opt/zenoss/ZenPacks/ZenPacks.zenoss.WindowsMonitor-
zenoss    5001  0.0  0.5 466484 96048 ?        S    08:45   0:07 /opt/zenoss/bin/python /opt/zenoss/ZenPacks/ZenPacks.zenoss.WindowsMonitor-
zenoss    5036  0.0  0.5 462340 94924 ?        S    08:45   0:01 /opt/zenoss/bin/python /opt/zenoss/ZenPacks/ZenPacks.zenoss.WindowsMonitor-
root      5048  0.0  0.0  74824  1156 ?        Ss   08:45   0:00 crond
xfs       5069  0.0  0.0  20096   980 ?        Ss   08:45   0:00 xfs -droppriv -daemon
root      5109  0.0  0.0   3808   488 tty1     Ss+  08:45   0:00 /sbin/mingetty tty1
root      5110  0.0  0.0   3808   492 tty2     Ss+  08:45   0:00 /sbin/mingetty tty2
root      5111  0.0  0.0   3808   488 tty3     Ss+  08:45   0:00 /sbin/mingetty tty3
root      5112  0.0  0.0   3808   492 tty4     Ss+  08:45   0:00 /sbin/mingetty tty4
root      5113  0.0  0.0   3808   492 tty5     Ss+  08:45   0:00 /sbin/mingetty tty5
root      5114  0.0  0.0   3808   492 tty6     Ss+  08:45   0:00 /sbin/mingetty tty6
root      5128  0.0  0.0 256776 15504 ?        SN   08:46   0:00 /usr/bin/python -tt /usr/sbin/yum-updatesd
root      5130  0.0  0.0  12932  1116 ?        SN   08:46   0:00 /usr/libexec/gam_server
root      5137  0.0  0.0  90156  3716 ?        Rs   08:46   0:01 sshd: ***@pts/0
root      5139  0.0  0.0  66204  1648 pts/0    Ss   08:46   0:00 -bash
zenoss    5219 26.6 35.3 6161712 5810148 ?     R    08:48  39:13 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zentrap.py --configfi
zenoss   10414  3.0  0.8 492300 132808 ?       S    11:07   0:16 /opt/zenoss/bin/python /opt/zenoss/Products/ZenHub/zenhubworker.py --config
zenoss   10619  6.6  0.7 479924 120544 ?       S    11:13   0:10 /opt/zenoss/bin/python /opt/zenoss/Products/ZenHub/zenhubworker.py --config
root     10846  1.0  0.0  50068  5768 ?        S    11:15   0:00 /opt/zenoss/bin/nmap -iL /tmp/zenping_nmap_f8e3rK -sn -PE -n --privileged -
root     10850  0.0  0.0  65608   984 pts/0    R+   11:15   0:00 ps aux
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70917#70917]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
hydruid
2013-01-10 16:38:26 UTC
Permalink
hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70933#70933

--------------------------------------------------------------
It depends on how many devices you are monitoring, but that definitely
seems excessive to me. Unfortunately I don't have enough experience with
this type of issue to help. The IRC channel on freenode would be your best
bet for an immediate response!

Hydruid
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70933#70933]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Philip Warren
2013-01-10 16:59:20 UTC
Permalink
Philip Warren [http://community.zenoss.org/people/pwarren] created the discussion

"Re: Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70935#70935

--------------------------------------------------------------
You are likely encountering this issue: http://jira.zenoss.com/jira/browse/ZEN-3005 http://jira.zenoss.com/jira/browse/ZEN-3005

This problem has been resolved in the latest 4.2.3 release of Zenoss.

Thanks,

Philip
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70935#70935]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Justin Simmons
2013-01-11 13:30:12 UTC
Permalink
Justin Simmons [http://community.zenoss.org/people/jmsimmons] created the discussion

"Re: Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70926#70926

--------------------------------------------------------------
Yup - the 4.2.3 install fixed all my issues. Appreciate the help very much!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70926#70926]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
hydruid
2013-01-11 16:42:41 UTC
Permalink
hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: Zenoss 4.2 RAM Utilization/Other Issues"

To view the discussion, visit: http://community.zenoss.org/message/70948#70948

--------------------------------------------------------------
Awesome! Glad to hear everything is worked out :)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70948#70948]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Loading...