Getting more performance

[will ton]

will ton [http://community.zenoss.org/people/wton] created the discussion

"Getting more performance"

To view the discussion, visit: http://community.zenoss.org/message/69979#69979

--------------------------------------------------------------
I'm relatively new to Zenoss and trying to understand how I can get Zenoss to push more IO.  I have a very fast storage storing all of the RRD files, but there seemes to be a bottleneck somewhere that isn't allowing updates to push the storage very hard.  I am using rrdpound to simulate a workload and when initially creating the RRD files I can push 200K IOPS and see that the queue on the disk is very high.  Once updates kick in IO performance drops down to 10K IOPS and doesn't seem like the process is doing much. 

I tried increasing the number of writer threads for rrdcached, but that didn't do much.  Running "iotop" I see that each rrdcached worker is doing very little work and CPU is only at about 10%.  Doing an strace on one of the rrdcached processes shows that it isn't spending a whole lot of time doing open's or writes.  Any other things that I could try to tune it to go faster?

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
96.68    3.498010         751      4660       374 futex
  1.98    0.071752          12      6180           madvise
  0.61    0.021993          21      1030           munmap
  0.40    0.014342          14      1030           msync
  0.29    0.010618          10      1030           mmap
  0.02    0.000856          86        10           mprotect
  0.00    0.000172           0      1030           fstat
  0.00    0.000126           0      1030           open
  0.00    0.000118           0      1030           close
  0.00    0.000106           0      1030           fcntl
  0.00    0.000073           0      1030           fadvise64
  0.00    0.000000           0        11           write
------ ----------- ----------- --------- --------- ----------------
100.00    3.618166                 19101       374 total
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69979#69979]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Ryan Matte]

Ryan Matte [http://community.zenoss.org/people/rmatte] created the discussion

"Re: Getting more performance"

To view the discussion, visit: http://community.zenoss.org/message/70114#70114

--------------------------------------------------------------
Zenoss uses the zenrrdcached daemon.  It caches RRD writes and only periodically writes back to storage.  This is to keep from destroying IO on servers with slower storage.  I don't know why you would actually want RRD hammering your storage?  That really makes no sense to me.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70114#70114]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[nilie]

nilie [http://community.zenoss.org/people/nilie] created the discussion

"Re: Getting more performance"

To view the discussion, visit: http://community.zenoss.org/message/70178#70178

--------------------------------------------------------------
Ryan,

My production Zenoss Core v3 server is showing an average of 21% time waiting for IO so I would like to know :
1. Is it possible to use rrdcached with Zenoss Core v3.1 ?
2. In case it is possible, would it be of any help ?

Thanks
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70178#70178]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Ryan Matte]

Ryan Matte [http://community.zenoss.org/people/rmatte] created the discussion

"Re: Getting more performance"

To view the discussion, visit: http://community.zenoss.org/message/70179#70179

--------------------------------------------------------------
Core v3 is not designed to use rrdcached.  You could hack it in but it would take a lot of time and wouldn't be pretty.  If you were to upgrade to Core v4 rrdcached would certainly help with the disk IO, but there would be higher memory requirements (RAM).
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70179#70179]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]