Linux memory thresholds

[ebogaard]

ebogaard [http://community.zenoss.org/people/ebogaard] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65423#65423

--------------------------------------------------------------
Sorry for opening this thread again, but to me it seems this problem still isn't solved: Zenoss cannot monitor the real memory usage of a Linux installation. This is really annoying.

Even though it is quite easy to with a formula like this (assuming 90% memory usage is your threshold):
memAvailReal + memBuffer + memCached > memTotalReal * 0,1

The problem here is the left side of this fomula: is there any way to combine several datapoints in an alias or something like that?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65423#65423]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James Stewart]

James Stewart [http://community.zenoss.org/people/amorphic] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65496#65496

--------------------------------------------------------------
ebogaard...

I looked into this fairly extensively and from what I can gather, there is no way for the zenoss snmp performance engine to perform operations across values obtained by polling multiple OIDs. As such, there's no out-of-the box way to perform the required equation.

In the end, I wrote a python script that collects the various values via snmp and derives the actual memory usage. I use a command data source to run this against my linux servers.

If it would be useful I can create a zenpack that pulls together my script, the command data source and the graph output?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65496#65496]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ebogaard]

ebogaard [http://community.zenoss.org/people/ebogaard] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65522#65522

--------------------------------------------------------------
James, Shane, thanks for you suggestions!

James, if you have a script in place and can make a Zenpack from it, that would be great.
Also checked out the 'Forumla Data Source' from Ryan Mattes that Shane suggested and that looks like a good suggestion as well, although this might be hard on the Zenoss server.

I'll try both.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65522#65522]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[linuxdynasty]

linuxdynasty [http://community.zenoss.org/people/linuxdynasty] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65538#65538

--------------------------------------------------------------
You can use my script, for monitoring Memory or swap percentage used...
Example how to use the script below...
***@localhost:/usr/local/zenoss/zenoss/libexec$ python getMemPercentage.py -d server1
OK |swapPercentageAvail=96 swapPercentageUsed=3 memPercentageAvail=30 memPercentageUsed=69


https://github.com/linuxdynasty/Linuxdynasty/blob/de98e1b4d0946bd7c569d1f6952aaa926f4df032/asanabria/Scripts/python/getMemPercentage.py https://github.com/linuxdynasty/Linuxdynasty/blob/de98e1b4d0946bd7c569d1f6952aaa926f4df032/asanabria/Scripts/python/getMemPercentage.py
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65538#65538]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ebogaard]

ebogaard [http://community.zenoss.org/people/ebogaard] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65573#65573

--------------------------------------------------------------
Thanks for your script. Will try it out!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65573#65573]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ebogaard]

ebogaard [http://community.zenoss.org/people/ebogaard] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65596#65596

--------------------------------------------------------------
Tried the script, but this isn't really what I'm searching for. It still doesn't subtract the cache and buffers in the memory from the total used memory (or adds both tot the free memory).
And it seems to be working quite slowly and take up quite some memory, so probably isn't for a production environment with multiple monitoring subjects.

Thanks anyway!
James, as you have a script that derives the actual usage, could you make it public?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65596#65596]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65598#65598

--------------------------------------------------------------
ebogaard:

Once you find a script that produces the result you want I highly reccomend you dump the value in to a custom net-snmp oid and poll the oid using zenperfsnmp in zenoss. Command datasources don't scale well in some cases.

Best,
--Shane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65598#65598]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65512#65512

--------------------------------------------------------------
ebogaard:

You can do this one of three ways.

1) Ryan Mattes comibined datasource zenpack. Don't use this in medium or larger scenarios.
2) Write a command datasource that fetches the values and adds them as needed
3) Write a script that gets the added values and puts them in a custom oid. Poll this oid.

Best,
--Shane (Hackman238)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65512#65512]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ebogaard]

ebogaard [http://community.zenoss.org/people/ebogaard] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65640#65640

--------------------------------------------------------------
Found some time today, and in hindsight, it wasn't all that difficult to gat the 'real' free memory. I did the following:
1. Made a script 'memoryFreee.sh' that uses 'free' to get the real free memory (in KB):
/usr/bin/free -k | awk '/cache:/ {print $2,$3,$4}' | sed 'N;s/\n/ /' | awk '{print ($3)}'

2. Extended snmp with this script. In snmpd.conf:
extend memoryFree /bin/sh /opt/scripts/memoryFree.sh

3. Added the resulting OID. to zenoss and set an threashold:
1.3.6.1.4.1.8072.1.3.2.4.1.2.10.109.101.109.111.114.121.70.114.101.101.1
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65640#65640]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/65657#65657

--------------------------------------------------------------
ebogarrd:

Excellent!

Best,
--Shane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65657#65657]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[felipe.castelar]

felipe.castelar [http://community.zenoss.org/people/felipe.castelar] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/69537#69537

--------------------------------------------------------------
I've implemented the Memory Threshold quite the same way as you did. Here I called it "Mem Utilization".

It's working pretty cool here ( >90% -> Event). I'm getting the message "threshold of Mem Utilization not met: current value:xxxx" associated with the /Perf/Memory event class on my event console.

Now I having difficulties transforming the /Perf/Memory event. I've created the defaultmapping on the /Perf/Memory event class and used the Rule evt.summary.find('threshold of Mem Utilization not met')>-1 and then the Transform evt.summary="my test text".

How have you implemented the transform you've just described above? Have you  created a mapping? Which Rule or Event Class Key have you used?

Thanks in advance,

Felipe Castelar
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69537#69537]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chris Smith]

Chris Smith [http://community.zenoss.org/people/csmith] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/73986#73986

--------------------------------------------------------------
Hoping to get some help on the tips above as well. Thanks in advance :)

here.hw.totalMemory/1024 * 0.15 seems to work but I know it's not accurate.

(here.hw.totalMemory/1024 * 0.95) - ((here.getRRDValue('memBuffer') or 9e9) + (here.getRRDValue('memCached') or 9e9)) does not work for me. My Solaris box is using 22% memory and I set it to 0.95 as shown above...it never triggered.


The transform didn't seem to work at all. I placed he transform under Events > Perf > Memory > Transform and saved it. I'm still getting alarms like "threshold of High Memory Utilization not met: current value 1629304.000000"
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73986#73986]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74006#74006

--------------------------------------------------------------
Chris,

I would highly reccomend avoiding using here.getRRDValue, it really won't scale well.

Figuring the proper free memory can be best done using https://github.com/zenoss/ZenPacks.zenoss.CalculatedPerformance https://github.com/zenoss/ZenPacks.zenoss.CalculatedPerformance

My recently released Improved Linux Monitor ZenPack has a good example of how to do this for calculating free memory. https://github.com/Hackman238/ZenPacks.Rackspace.LinuxMonitor https://github.com/Hackman238/ZenPacks.Rackspace.LinuxMonitor

In the image below a new datapoint representing (((Free + Cached) - Buffers) * 1024) has been created and is represented by green. A simple Min/Max threshold can be set against that datapoint using ((here.hw.totalMemory/1024) * 0.95) or similar. In my ZenPack I've added a zProperty where one can dictate a percentage of free memory, below which a threshold is event is triggered. That threshold is represented by ((here.hw.totalMemory) * here.getFreeMemoryThresholdPercentLimit()) Helps simplify management.
Loading Image... Loading Image...

This wil not only scale better, but will also simplify reporting on historical freeMemory (less calculations).

Best,
--Shane Scott (Hackman238)
http://shanewilliamscott.com http://shanewilliamscott.com
http://linkedin.com/in/shanewilliamscott http://linkedin.com/in/shanewilliamscott
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74006#74006]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chris Smith]

Chris Smith [http://community.zenoss.org/people/csmith] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74028#74028

--------------------------------------------------------------
You rock Shane. I will do my best to get this implemented and get back with questions. Do you have your alarms transformed to percentage or just using the raw value?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74028#74028]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74070#74070

--------------------------------------------------------------
Chris,

Thanks. I didn't include transforms since many people have their own transforms and it would complicate the install. I can offer you what we use if you're interested.

Best,
--Shane Scott (Hackman238)
http://shanewilliamscott.com http://shanewilliamscott.com
http://linkedin.com/in/shanewilliamscott http://linkedin.com/in/shanewilliamscott
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74070#74070]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chris Smith]

Chris Smith [http://community.zenoss.org/people/csmith] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74059#74059

--------------------------------------------------------------
I would not mind seeing them for reference. I have the general idea down and I'm able to decipher some python but most of the time I cannot get most transforms I find on the forums to function correctly.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74059#74059]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[barney34]

barney34 [http://community.zenoss.org/people/barney34] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74563#74563

--------------------------------------------------------------
Scott,

Where is the EGG files I can not see them on the Wiki or github.   Thank you kindly sir.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74563#74563]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74565#74565

--------------------------------------------------------------
Barney,

https://github.com/Hackman238/ZenPacks.Rackspace.LinuxMonitor/raw/master/ZenPacks.Rackspace.LinuxMonitor-1.0.0-py2.7.egg https://github.com/Hackman238/ZenPacks.Rackspace.LinuxMonitor/raw/master/ZenPacks.Rackspace.LinuxMonitor-1.0.0-py2.7.egg

Best,

--Shane Scott (Hackman238)
http://shanewilliamscott.com http://shanewilliamscott.com
http://linkedin.com/in/shanewilliamscott http://linkedin.com/in/shanewilliamscott
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74565#74565]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ynodelman]

ynodelman [http://community.zenoss.org/people/ynodelman] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74758#74758

--------------------------------------------------------------
Hi Shane,

Would you advise me on where I can download realFreeMemory.pl script? Furthermore, is there a special setup for monitoring individual CPU Cores?
-----
Preparing Command...
Executing command $ZENHOME/libexec/realFreeMemory.pl [args omitted] against servername
/bin/sh: /opt/zenoss/libexec/realFreeMemory.pl: No such file or directory
DONE in 0 seconds
--------
Thank you.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74758#74758]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Shane Scott]

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/74761#74761

--------------------------------------------------------------
ynodelman,

No idea. That script isn't part of or used in my LinuxMonitor pack. It's enytirely snmp and perfcalc datasources.

Best,
--Shane Scott
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74761#74761]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[John Komara]

John Komara [http://community.zenoss.org/people/koka424] created the discussion

"Re: Linux memory thresholds"

To view the discussion, visit: http://community.zenoss.org/message/75197#75197

--------------------------------------------------------------
I installed ZenPacks.zenoss.CalculatedPerformance-1.0.7.egg and ZenPacks.Rackspace.LinuxMonitor-1.0.0-py2.7.egg zenpacks. I started to recieve a transform error and tracked it down to a server that was reporting high memory buffer usage. The formula that came with the zenpack for calcFreeMemory was ((memAvailReal + memCached) - memBuffer) *1024. memBuffer was bigger than memAvailReal + memCached. This was sending a negative value to the transform and causing an error.

I changed the formula to (memAvailReal + memCached + memBuffer) and my free memory now matches up with what is reported by "free -m" on my devices. Am I missing something with the ((memAvailReal + memCached) - memBuffer) formula or was this just an error?

Thanks.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75197#75197]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]