Yet Another WMI Question...

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69798#69798

--------------------------------------------------------------
Hi folks.  Just about done pulling my hair out here with Zenoss and WMI. I've been up and down every post I can find, coming tantalizingly close to solving things but never quite getting there.  I've created a testbed environment in order to isolate as much as I can.  Here's what I've got so far.

* One Windows 2008 R2 EC2 instance (not attached to a domain).  Call it TARGET.
* Windows Firewall is off
* EC2 Firewall allows ALL TRAFFIC

* Another Windows 2008 R2 EC2 instance, also standalone (no domain).  Call it SOURCE
* WBEMTest and Solar Windows WMI Monitor on SOURCE are *both able to connect* to TARGET using admin-level credentials.  I can query against WMI without issue. WMI is definitely working on TARGET.

* I have a Centos 6 instance running Zenoss Core 4.  Call this ZENOSS.  Again, completely standalone, isolated system.
* Running the following wmic command (and many, many variations) yields...

./wmic -d20 -U ./Administrator%Passw0rd //ec2-12-42-97-33.compute-1.amazonaws.com "select LoadPercentage from Win32_Processor"

.... some debug output... about a 120 second pause.. and finally..

[librpc/rpc/dcerpc_connect.c:337:dcerpc_pipe_connect_ncacn_ip_tcp_recv()] failed NT status (c0000017) in dcerpc_pipe_connect_ncacn_ip_tcp_recv
[librpc/rpc/dcerpc_connect.c:820:dcerpc_pipe_connect_b_recv()] ENTER function dcerpc_pipe_connect_b_recv
[librpc/rpc/dcerpc_connect.c:828:dcerpc_pipe_connect_b_recv()] failed NT status (c0000017) in dcerpc_pipe_connect_b_recv
[librpc/rpc/dcerpc_connect.c:837:dcerpc_pipe_connect_b_recv()] EXIT  function dcerpc_pipe_connect_b_recv (PASS)
[lib/com/dcom/main.c:1054:bind_new_pipe_continue()] Unable to bind to 2001:0:9d38:6ab8:243a:35ef:f53c:a010[49154]: NT_STATUS_NO_MEMORY
[lib/com/dcom/main.c:1098:try_next_binding()] dcom_get_pipe: Skipping binding \\\\IP-0AC35FEF[\\PIPE\\srvsvc]
[wmi/wmic.c:196:main()] ERROR: Login to remote object.
NTSTATUS: NT code 0xc002001b - NT code 0xc002001b



When I see network-sensitive apps pause, I immediately think name resolution problems. Now here's something interesting.  If I look at the wmic debug output I see...

[lib/com/dcom/main.c:1054:bind_new_pipe_continue()] Unable to bind to ip-0AC35FEF[49154]: NT_STATUS_IO_TIMEOUT
[lib/com/dcom/main.c:1054:bind_new_pipe_continue()] Unable to bind to 10.195.95.219[49154]: NT_STATUS_IO_TIMEOUT
[lib/util/util.c:334:interpret_addr()] sys_gethostbyname: Unknown host. 2001:0:9d38:6ab8:243a:35ef:f53c:a020

These are all valid interfaces on TARGET, but obviously they won't resolve/bind, because we're going across the WAN.  Is this relevant/bad/a glimmer of hope? I'm thinking that a) I should see a bind against the WAN address and b) all of these bad binds are timing out and perhaps taking the whole wmic attempt down with it.

Please.. please..  I love SNMP, but damnit Jim, I need more metrics  (see what I did there?  Bones and Scotty, all in one).

Please pepper me with questions and insults!

Thanks,
Mike
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69798#69798]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Doug Syer]

Doug Syer [http://community.zenoss.org/people/dsyer%40nwnit.com] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69815#69815

--------------------------------------------------------------
Hi Mike,

Are you natting between the collector/single zenoss instance and the target windows server?

If so, you will probably need to add the windows host names to dns even if you are connecting via ip address. Doesnt make alot of sense at face value I know.   To test add the dns name of the server you are polling to the local hosts file.  Not sure if it needs to be the FQDN or not.

If you arent natting between the collector and the windows host you dont need dns.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69815#69815]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69799#69799

--------------------------------------------------------------
No NAT involved.  Both TARGET and ZENOSS (and SOURCE) are EC2 instances.

I don't know if all the B-class and netbios name resolution failures evident in the debug logs are relevant or are just red herrings, but I thought it interesting.

There is nothing different between ZENOSS and SOURCE in terms of network position  (both are standalone EC2 instances), and WMI works SOURCE -> TARGET but does not work ZENOSS -> TARGET.  This tells me..

1) It's a problem with how ZENOSS (wmic) does WMI when going across the wan.
2) It's a problem with TCP callbacks (if they even happen) between TARGET and ZENOSS (though iptables is currently off on ZENOSS, and the EC2 firewall for ZENOSS allows every port/protocol, just like SOURCE and TARGET).
3) Whatever else I don't know enough to know about.  :)

Thanks.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69799#69799]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69800#69800

--------------------------------------------------------------
Still pouring through the debug logs. 


[lib/com/dcom/main.c:570:complete_activation()] Negotiated COM version: 5.1 using binding ncacn_ip_tcp:54.24.96.3[135]

This tells me (I believe) that it was successful at binding to tcp/135 against the TARGET wan IP address. Great! Unfortunately I then see all those Class-B/NetBios-based bindings I mention above...  to the 10.x.x.x address, netbios name, IPV6, etc, etc... none of which would work.  Why doesn't it try to bind using the public IP it clearly knew about when it bound to port 135??

[lib/com/dcom/main.c:1054:bind_new_pipe_continue()] Unable to bind to ip-0AC35FEF[49154]: NT_STATUS_NO_MEMORY
[lib/com/dcom/main.c:1054:bind_new_pipe_continue()] Unable to bind to 10.195.95.239[49154]: NT_STATUS_IO_TIMEOUT
[lib/com/dcom/main.c:1054:bind_new_pipe_continue()] Unable to bind to 2001:0:9d38:6ab8:243a:35ef:f53c:a010[49154]: NT_STATUS_NO_MEMORY


<sigh>
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69800#69800]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69820#69820

--------------------------------------------------------------
Update:  I set up the winexe service on TARGET, and I'm able to make winexe calls from ZENOSS -> TARGET without issue.

This is using the identical command I do for wmic (calling winexe and using a console command instead of a wbem query, of course).

So negotiation/operation of winexe is working.  Doesn't help the issues with wmic, but it's interesting.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69820#69820]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

This post might be inappropriate. Click to display it.

[Doug Syer]

Doug Syer [http://community.zenoss.org/people/dsyer%40nwnit.com] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69825#69825

--------------------------------------------------------------
Im pretty certain what you are seeing is a sucessful handshake on the microsoft rpc listener service on port 135.  Then the server isnt or cant send the results back on a high port.  Ive seen the same thing a few times when trying to query wmi using wmic or zen enterprises wmi modeler or pollers.

Ome time the issue was an application inspection issue on the firewall where it was throwing out the high port responses.  We also had to fix it when natting on e via setting dns on one or both sides, forget the details on that fix.

I know you can hard code the response port on the windows server side but thats not ideal.  Are you sure that there is no difference in application filtering on the two firewalls? 

Ive also seen it once where the wmi was loaded but dts wasnt.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69825#69825]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69828#69828

--------------------------------------------------------------
Turned out to be an incredibly annoying case of wmic iterating through all of the internal interfaces on TARGET and trying to bind to each of them for some follow-up connection (probably the asynchronous calback). 

Why wmic doesn't use the name it's originally handed to do the resolution, I don't know.  But once I put the netbios name of TARGET into ZENOSS's hosts file, wmic and Zenoss began behaving.

Thanks,
Mike
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69828#69828]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Doug Syer]

Doug Syer [http://community.zenoss.org/people/dsyer%40nwnit.com] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69830#69830

--------------------------------------------------------------
Yeah thats what fixed it the last time i saw it. 
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69830#69830]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69824#69824

--------------------------------------------------------------
FINALLY. Based on the debug line below, I added ip-0AC35XXX directly to my hosts file for TARGETs public IP.  Once I did that, wmic worked (and Zenoss also seems to be able to model the machine, though it's complaining about Process inventory not being supported).

I have no idea why wmic doesn't try binding to the public IP first (or ever?). Is there some way to tell wmic/Zenoss to resolve sensibly?  Maintaining the hosts file isn't really something that should have to be done.

Thanks.


[lib/com/dcom/main.c:1113:try_next_binding()] dcom_get_pipe: Trying binding ip-0AC35XXX[49154]
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69824#69824]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Lokisite]

Lokisite [http://community.zenoss.org/people/Lokisite] created the discussion

"Re: Yet Another WMI Question..."

To view the discussion, visit: http://community.zenoss.org/message/69834#69834

--------------------------------------------------------------
For those playing along at home, the problem is located in the dcom library.  You can see it here.

http://dev.zenoss.com/trac/changeset/7575 http://dev.zenoss.com/trac/changeset/7575

The callback iterates through the interface list only.  If the public IP isn't on the interface list, wmic will fail.  (Host entries mitigate this, but are a kludgy fix).  The dcom connection attempts should try the host value first, before the enumerated interface list.  At some point I'll rebuild the dcom lib and give it a shot.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69834#69834]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]