Alexander Fortin
2012-03-29 11:10:28 UTC
Alexander Fortin [http://community.zenoss.org/people/shaftoe] created the discussion
"Zencommand file descriptors grow indefinitely"
To view the discussion, visit: http://community.zenoss.org/message/65602#65602
--------------------------------------------------------------
Hi folks.
In a couple of months we've hit a couple of times the maximum number of file descriptors allowed for the zencommand process (1024 on our platform).
When this happens, it's a big problem: zencommand doesn't crash, just writes "Too many open files" exceptions in the logfile, and Zenoss other dameons don't notis this situation, so the result is that, according to the web frontend, everything is fine, but the truth is that no data / event for checks leveraging zencommand is actually working.
I've started to analyze a little what's causing this. First thing, I'm using zencommmand itself to check how many file descriptors is using, to get an idea of the growing trend. Here you see the latest days:
Loading Image...
Loading Image...
Note that the number of monitored devices is almost always the same ( more or less 30 boxes with some autoscaling ). At a firts sight, seems that the file descriptors get used for TCP connections, and I can actually see many ssh established connections to the same device (I thought that zencommand used just one connection to pipe every command, but I guess I was wrong).
So far I can't think a solution other than restarting the zencommand process once in a while, but of course if you can suggest a cleaner approch it will be very appreciated :)
--------------------------------------------------------------
Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65602#65602]
Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
"Zencommand file descriptors grow indefinitely"
To view the discussion, visit: http://community.zenoss.org/message/65602#65602
--------------------------------------------------------------
Hi folks.
In a couple of months we've hit a couple of times the maximum number of file descriptors allowed for the zencommand process (1024 on our platform).
When this happens, it's a big problem: zencommand doesn't crash, just writes "Too many open files" exceptions in the logfile, and Zenoss other dameons don't notis this situation, so the result is that, according to the web frontend, everything is fine, but the truth is that no data / event for checks leveraging zencommand is actually working.
I've started to analyze a little what's causing this. First thing, I'm using zencommmand itself to check how many file descriptors is using, to get an idea of the growing trend. Here you see the latest days:
Loading Image...
Loading Image...
Note that the number of monitored devices is almost always the same ( more or less 30 boxes with some autoscaling ). At a firts sight, seems that the file descriptors get used for TCP connections, and I can actually see many ssh established connections to the same device (I thought that zencommand used just one connection to pipe every command, but I guess I was wrong).
So far I can't think a solution other than restarting the zencommand process once in a while, but of course if you can suggest a cleaner approch it will be very appreciated :)
--------------------------------------------------------------
Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65602#65602]
Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]