Question : SQL 2005 resource group fails over unexpectedly in MS Cluster

We are seeing issues with our SQL cluster where the SQL 2005 cluster group fails over to the other node in the cluster.  It gives errors about the network name no longer being unavailable and communication link failure.  Below are the relevant cluster.log entries:

0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] printODBCError: sqlstate = 08S01; native error = 40; message = [Microsoft][SQL Native Client]Named Pipes Provider: The specified network name is no longer available.

0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] printODBCError: sqlstate = 08S01; native error = 40; message = [Microsoft][SQL Native Client]Communication link failure
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] OnlineThread: QP is not online.
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
0000108c.0000a108::2008/07/15-02:48:27.812 ERR  SQL Server : [sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure
000007c8.000010a4::2008/07/15-02:48:30.672 INFO [FM] NotifyCallBackRoutine: enqueuing event
000007c8.000010a4::2008/07/15-02:48:30.672 INFO [FM] Calling RmNotifyChanges in monitor 108c.
000007c8.00000984::2008/07/15-02:48:30.672 INFO [CP] CppResourceNotify for resource SQL Server
000007c8.00000984::2008/07/15-02:48:30.672 INFO [CP] CppRundownCheckpoints removing RNB for Software\Microsoft\Microsoft SQL Server\MSSQL.1\SQLServerSCP
000007c8.00000984::2008/07/15-02:48:30.687 INFO [CP] CppRundownCheckpoints removing RNB for Software\Microsoft\Microsoft SQL Server\MSSQL.1\PROVIDERS
000007c8.00000984::2008/07/15-02:48:30.703 INFO [CP] CppRundownCheckpoints removing RNB for Software\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLSERVER
000007c8.00000984::2008/07/15-02:48:30.703 INFO [CP] CppRundownCheckpoints removing RNB for Software\Microsoft\Microsoft SQL Server\MSSQL.1\Cluster
000007c8.00000984::2008/07/15-02:48:30.718 INFO [CP] CppRundownCheckpoints removing RNB for Software\Microsoft\Microsoft SQL Server\MSSQL.1\SQLserverAgent
000007c8.00000984::2008/07/15-02:48:30.718 INFO [CP] CppRundownCheckpoints removing RNB for Software\Microsoft\Microsoft SQL Server\MSSQL.1\Replication
000007c8.00000984::2008/07/15-02:48:30.734 INFO [CP] CppRundownCheckpoints removing RNB for Software\Microsoft\Microsoft SQL Server\MSSQL.1\CPE
000007c8.000009a8::2008/07/15-02:48:30.750 WARN [FM] FmpHandleResourceTransition: Resource Name = 1d3ab5eb-38e6-44da-8341-347b2b4aad30 [SQL Server] old state=2 new state=4
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumSendUpdate:  Locker waiting            type 0 context 8
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] Thread 0x9a8 UpdateLock wait on Type 0
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 1
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumpDoLockingUpdate successful, Sequence=16695 Generation=0
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumSendUpdate: Locker dispatching seq 16695      type 0 context 8
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumSendUpdate: Dispatching seq 16695      type 0 context 8 to node 2
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumSendUpdate: Locker updating seq 16695      type 0 context 8
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [GUM] GumSendUpdate: completed update seq 16695      type 0 context 8
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [FM] FmpPropagateResourceState: resource 1d3ab5eb-38e6-44da-8341-347b2b4aad30 failed event.
000007c8.000009a8::2008/07/15-02:48:30.750 INFO [FM] FmpHandleResourceFailure: taking resource 1d3ab5eb-38e6-44da-8341-347b2b4aad30 and dependents offline

Some details about the cluster setup:

* Two node HP BL685c G1 systems
* Windows 2003 x64 EE R2 SP1
* Heartbeat is a single nic on each node
* Public interface is teamed set to auto on each nic in the team
* Running SQL 2000 in its own cluster group and SQL 2005 in its own as well (only have issues with 2005 instance)

What could be causing this?  I am leaning towards possible nic issues but need assistance on further troubleshooting this issue.

Thank you.

Answer : SQL 2005 resource group fails over unexpectedly in MS Cluster

Hi

MSCS is notoriously reliant on network stability and latency.

Try the following tests:
- running the teamed NICs in TLB or active/passive mode
- setting the teamed NICs to public only usage
and see if the problem clears up

If it persists, there could be other network issues (VLAN flapping etc etc) further downstream in your topology. Have you checked matched up with a known good working configuration for the exact switch and NIC firmware levels? It makes a significant difference.

In that case, you might consider unbinding the team and running just one NIC dedicated for public, so you have PUB HB1 HB2 on each machine.

We have seen similar behaviour on BL465c and BL485cs using teamed NICs configured for LACP, and we had to revert to an unteamed configuration.