Infrastructure Page slow to load

[tcaiazza]

tcaiazza [http://community.zenoss.org/people/tcaiazza] created the discussion

"Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/63981#63981

--------------------------------------------------------------
I'm running zenoss 3.2.1, and when I click the infrastructure page it take a very long time to load(10minutes+), some times it doesn't even load.

I found this old note in a thread somewhere, http://community.zenoss.org/docs/DOC-7798#d0e422 http://community.zenoss.org/docs/DOC-7798#d0e422,  about applying a patch for slow loading trees.  This was back in 3.0.1, so I'm assuming that this was fixed in 3.2.1.  Is that correct?

I don't see anything out of the ordinary on the server and restarting doesn't seem to help.  The logs look fine, granted I'm not sure if I'm looking at the right ones just what is under $ZENHOME/log.  Has anyone had issues like this?  Or can anyone point me to an area to focus on?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63981#63981]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[tcaiazza]

tcaiazza [http://community.zenoss.org/people/tcaiazza] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64019#64019

--------------------------------------------------------------
Ok, so this doesn't seem to be an issue with my server.  It appears that the infrastructure page is slow to load anytime I have over 50 items to display at once.  I've tried this is both chrome, and firefox.  Has anyone else seen this before?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64019#64019]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Stefan Reinke]

Stefan Reinke [http://community.zenoss.org/people/Stefan] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64023#64023

--------------------------------------------------------------
Hi,

I am monitoring over 200 devices and my page takes about 5-15 secondes to load. Using Zenoss 3.2.1 and Chrome.

What type of devices are you monitoring ?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64023#64023]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[tcaiazza]

tcaiazza [http://community.zenoss.org/people/tcaiazza] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64029#64029

--------------------------------------------------------------
I'm monitoring mostly routers/switches, but none of the backend process, like zenperfsnmp, are having issues.  So this looks purely to be an issue with the web gui.

Stefan, does the infrastructure page load fine when you display 50+ items at once?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64029#64029]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Stefan Reinke]

Stefan Reinke [http://community.zenoss.org/people/Stefan] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64030#64030

--------------------------------------------------------------
Are all those devices pulling information (routes) etc. Or only some of them ? I have checked I only have about 30 devices pulling information and the rest only sends a icmp packet to check if the device is still alive.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64030#64030]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64031#64031

--------------------------------------------------------------
On 3.2.1 with Opera 11.60, the initial ~40 or so of 167 devices show up fine for me. It does take a bit (30 seconds) to scroll farther as it loads more... Is your delay with the initial loading or with scrolling down the list?

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64031#64031]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[tcaiazza]

tcaiazza [http://community.zenoss.org/people/tcaiazza] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64046#64046

--------------------------------------------------------------
Mine is on the initial load.  If I shrink my broswer window down a bit so less than 50 devices are being displayed, the page loads up almost instantly.

Can you guys try zooming out/shrinking you text size so the infrastructure page will show more than 50 devices at a time?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64046#64046]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64047#64047

--------------------------------------------------------------
tcaiazza:

This can be many things, however, it's commonly substantiated by a browser JS engine issue. Can you run the follwing benchmark and let me know what your browser scores?

http://v8.googlecode.com/svn/data/benchmarks/v5/run.html http://v8.googlecode.com/svn/data/benchmarks/v5/run.html

Best,
--Shane (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64047#64047]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64032#64032

--------------------------------------------------------------
You must have good eyes. I had to zoom out to 50% (where I can't read anything) to get more than 50 devices shown. It is indeed taking a while to display. My suggestion is "don't do that" for what it's worth. By default maximized web page window, I see 25 devices... Even up to 40 shows fast. It may well be a bug, but the work-around is easy.

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64032#64032]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[tcaiazza]

tcaiazza [http://community.zenoss.org/people/tcaiazza] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64048#64048

--------------------------------------------------------------
Shane - here are my scores
Score: 13512
Richards: 12130
DeltaBlue: 18629
Crypto: 12912
RayTrace: 17254
EarleyBoyer: 28671
RegExp: 2765
Splay: 20609

James -  I have a big monitor.  I'm fine with just making my broswer window smaller to make this work.  I just wanted to make sure it wasn't an issue with my zenoss setup.  But this does appear to be a bug that someone should look at.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64048#64048]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64049#64049

--------------------------------------------------------------
tcaiazza:

You might want to increase your zope object cache and zope cache (both in zope.conf) as well as ensure that mysql caching and threads are appropriate. The infrastructure page taxes the global catalog (in zope) and calls methods which query for event totals in mysql. If the page has to wait for either (due to IO wait or invalidation) the page will load very slowly. Maximizing caching helps ensure that this condition only occurs during invalidation events, which can be handled by increasing the invalidation entry in the zeo.conf.

What are the current values? How large is your installation?

--Shane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64049#64049]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[tcaiazza]

tcaiazza [http://community.zenoss.org/people/tcaiazza] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64051#64051

--------------------------------------------------------------
Yeah, at first I assumed this was an issue with performance of my zenoss server, so I went through everything here, http://community.zenoss.org/thread/14051 http://community.zenoss.org/thread/14051.  My cache-size is set to 50000. I've also took a close look at all the server IO and everything looked fine.

Also, I'm still running an older version of zenoss in production and the issues I've been having are on the box I'm mirgrating to, so the setup is very small only 53 devices right now.

Shane, did you try zooming out to see over 50 devices at once to see if you see the same issue I've been having?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64051#64051]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64052#64052

--------------------------------------------------------------
tcaiazza:

Yeah that should definately be fine. I did try to replicate the problem, however, I couldn't duplciate the issue. My systems have massive amounts of resources, cache virtually everything and load balance across 40 zopes and is therefore not a good analog for this.

It might be interesting to login to mysql and show processes while refreshing the page. I'm interested in the queury time for the page.

--Shane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64052#64052]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64033#64033

--------------------------------------------------------------
Zenoss graphing shows a MySQL  Spike when I tested this , FWIW...  Loading Image... Loading Image...
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64033#64033]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64053#64053

--------------------------------------------------------------
James:

Ouch! thats a serious spike.

Watching here we don't see more than a tiny ripple. Here's my test config:
[mysqld]
socket = /opt/zends/data/zends.sock
basedir = /opt/zends
datadir = /opt/zends/data
port = 13306
user = zenoss
innodb_file_per_table
skip_external_locking

key_buffer = 2048M                              # 25% of total memory on a machine that mainly runs MySQL is quite common
innodb_buffer_pool_size = 22528M                # The larger you set this value, the less disk I/O is needed to access data in tables
innodb_additional_mem_pool_size = 40M           # The more tables you have in your application, the more memory you need to allocate here
innodb_log_file_size = 512M                     # from 1MB to 1/N-th of the size of the buffer pool, where N is the number of log files in the group
innodb_log_buffer_size = 8M                     # allows large transactions to run without a need to write the log to disk before commit
innodb_flush_log_at_trx_commit = 2              # complicated, see the docs. 1 is safest
innodb_lock_wait_timeout = 50                   # timeout in seconds an InnoDB  transaction may wait for a row lock before giving up
max_allowed_packet = 64M                        # should be as big as the largest BLOB you want to use
table_cache = 24576                             # number of open tables for all threads
sort_buffer_size = 256M                         # Increase this value for faster ORDER BY or GROUP BY operations
read_buffer_size = 128M                         # If you do many sequential scans, you might want to increase this value
read_rnd_buffer_size = 16M                      # change the session variable only from within those clients that need to run large queries
myisam_sort_buffer_size = 1024M                 # size of the buffer that is allocated when sorting MyISAM indexes
thread_cache_size = 512                         # can be increased to improve performance if you have a lot of new connections
query_cache_size = 128M                         # amount of memory allocated for caching query results
query_cache_limit = 3072M                       # Don't cache results that are larger than this number of bytes, leave high for zenoss
thread_concurrency =24                          # Try number of CPU's*2 for thread_concurrency
innodb_flush_method=O_DIRECT                   # according to a DBA here are Rackspace, this setting is difficult to apply correctly

interactive_timeout = 600
wait_timeout = 86400
max_connections = 1500
max_user_connections = 1500
max_heap_table_size = 32768

[client]
socket = /opt/zends/data/zends.sock
user = zenoss

[mysql]
prompt = "zends> "
no-auto-rehash

[mysqldump]
quick
max_allowed_packet = 64M

[isamchk]
key_buffer = 1024M
sort_buffer_size = 256M
read_buffer = 256M
write_buffer = 256M

[myisamchk]
key_buffer = 1024M
sort_buffer_size = 256M
read_buffer = 256M
write_buffer = 256M
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64053#64053]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Infrastructure Page slow to load"

To view the discussion, visit: http://community.zenoss.org/message/64034#64034

--------------------------------------------------------------
I don't know that we've tuned anything here. I think we plan on increasing the RAM in the server at some point, so I might play with this when we're down for that.

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64034#64034]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]