Question : Multiple Errors on Exchange 2003 Server

We had an unknown incident around 1:00 AM Eastern, which effectively took our Exchange Server 2003 off the network.  Here are some of the repeated errors in the Application Log:

Event Type:      Error
Event Source:      MSExchangeSA
Event Category:      Monitoring
Event ID:      9099
Date:            6/4/2008
Time:            1:07:45 AM
User:            N/A
Computer:      EXCH1
Description:
The MAD Monitoring thread was unable to read the state of the services, error '0x80041001'.

Event Type:      Warning
Event Source:      PerfOS
Event Category:      None
Event ID:      2012
Date:            6/4/2008
Time:            1:08:02 AM
User:            N/A
Computer:      EXCH1
Description:
Unable to get system process information from system. The status code returned is in the first DWORD in the Data section.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 9a 00 00 c0               ?..À    

Event Type:      Error
Event Source:      MSExchangeDSAccess
Event Category:      Topology
Event ID:      2114
Date:            6/4/2008
Time:            1:10:38 AM
User:            N/A
Computer:      EXCH1
Description:
Process WMIPRVSE.EXE -EMBEDDING (PID=6564). Topology Discovery failed, error 0xffffffff.

Event Type:      Error
Event Source:      Perflib
Event Category:      None
Event ID:      1023
Date:            6/4/2008
Time:            1:10:50 AM
User:            N/A
Computer:      EXCH1
Description:
Windows cannot load extensible counter DLL Tcpip, the first DWORD in data section is the Windows error code.

Data:
0000: 5a 04 00 00               Z...    

Event Type:      Warning
Event Source:      MAR5Core
Event Category:      None
Event ID:      6702
Date:            6/4/2008
Time:            1:21:08 AM
User:            N/A
Computer:      EXCH1
Description:
Plugin SMTPSend reports exception.

SMTP Send: Connection to the server failed. Please check your SMTP server settings.
            
Additional info: Timeout.

Event Type:      Error
Event Source:      ESE BACKUP
Event Category:      Callback
Event ID:      904
Date:            6/4/2008
Time:            8:11:29 AM
User:            N/A
Computer:      EXCH1
Description:
Information Store (5828) Callback function call ErrESECBPrepareInstanceForBackup ended with error 0xC7FE1F45 Instance not found

The server never bluescreened or went completely down.  No one could make a new connection to Exchange, although some had connections still active in the morning that were made the previous evening.  I couldn't login to the console or RDP and when I tried to shutdown -r remotely, it reported the server was locked and the FORCE option must be used.  It still didn't respond to shutdown with -f so I had to take it down hard.  Fortunately, it seems to have recovered from that power cycle and is running now, but I'd like to get to the root cause.

What is most likely?  Rogue process?  Failing NIC?  Hard drive contention?  Has anyone seen these symptoms before?

Thanks for the help!
Glen

Answer : Multiple Errors on Exchange 2003 Server

It would appear that something ran on the server around 1:00am, this then had problems and resulted in the server falling over.

Check all the jobs and maintenace (including Backup, WSUS, Updates and Exchange Maintenance) processes that are scheduled to run at this time. It could be that you have too many running at the same time which are causing the server to fail - you may need to adjust the maintenance windows so the server does not get overloaded.

Also check that you server has sufficient RAM and Processing power to cope with the day to day tasks.