Often Exchange administrators will receive escalated help desk tickets from users complaining that Exchange is “slow” and demanding resolution. These sorts of tickets can be extremely challenging to work, since the subjective nature of slowness is often combined with an inability to replicate the problem, or the problem is intermittent. The Exchange admin can take a look at the server(s) for high CPU utilization, low memory conditions, disk and network queue lengths exceeding the norm, and finding nothing, shrug it back off to the desktop support team as a client issue.

While it is often a client issue, there are several places between Outlook and a user’s mailbox that can cause intermittent slowness, and are fair to call networking bottlenecks. In a six part series of articles, we’ll look at how Exchange interacts on the network with various other services to help you identify network issues, and troubleshoot them when they occur.

In many cases, troubleshooting Exchange network bottlenecks will require a network trace, and may also require performance monitor counters. This series of articles will talk about both of those in general terms; how to use NetMon or Wireshark, and PerfMon are out of scope. In Part 4 of this series, we’re going to look at the humble physical layer (DoD, not OSI) and discuss troubleshooting NICs.

NICs

We’re now down where the rubber meets the road, that is, where the packets meet the wire. Your Network Interface Cards can be the most important part of the entire network connectivity between client process and server process, and are also the most commonly overlooked aspect of the entire communications channel. I’ve seen many a case where Exchange network performance issues came down to problems with the NIC, but days had gone by troubleshooting the problem, or weeks just accepting the poor performance, before anyone thought to look at the NICs. If the NICs aren’t happy, ain’t nobody happy so let’s make sure those NICs smile.

The differences between the various physical connections are beyond the scope of this article, but the recommendations and troubleshooting suggestions in this article should apply equally to all types of NIC, whether copper or fibre based, and whether physical or virtual. Let’s start with some best practices for connecting up all your servers and clients:

Use quality NICs

There are times to save money, and there are times to spend the extra for the best, and as far as Exchange servers are concerned, you cannot go wrong spending a little extra on the higher quality NICs. Single port or multi-port, specific name brand not as important, but don’t buy the cheap one off NICs or limit yourself to what is built-in to your server.

Use good cables

I take pride in my ability to “roll my own” cables (Ethernet, not fibre-optic) and I also know that name-brand cables can cost a fortune, but here again is where you don’t want to take any chances. All of your drop cables should be commercially made, but at the same time, don’t assume that because they are, they are faultless. Make it a habit to test all cables early in the troubleshooting process if not at time of install.

Use quality, managed switches

Inexpensive unmanaged switches are good for home use, or to provide last minute patches in a meeting room without wireless, but have no place in a datacenter. Make sure all your servers directly connect to managed switches that can provide you details and statistics about the physical connection.

With that out of the way, now we’ll move on to some more best practices that should also be the second steps you take on the server when troubleshooting connectivity issues, right after reseating all the cables.

Hardware Drivers

Make absolutely certain you are running the latest hardware drivers. Check the vendor site, and read the documentation for any known issues that might correlate to your problem, but unless there is something contraindicated in that documentation, make sure you have the latest supported drivers. If you do though, consider downgrading one rev just in case you have encountered a new bug.

Firmware

Don’t just stop at the software drivers for your NICs, make sure you have the latest firmware installed as well.

TCPIP.SYS

Check the Microsoft operating system drivers for your specific platform, and if you are not running the latest TCPIP driver, upgrade immediately. I have personally seen dozens of problems magically disappear just by catching up on patches. Of course, I do recommend staying current on all patches, but this is one that should have no exceptions.

Teaming

More connectivity problems have been “solved” by “breaking the team” than any other single fix in history. If you have having network connectivity problems and are using network teaming, break the team and see if the problem goes away. Do this early on, as it is a quick thing to check, and to put back if that is not the problem. Odds are that it is, and in that case, you need to troubleshoot network teaming, not Exchange networking. The solution will usually be with updating drivers, fixing a problem with your configuration, or something on the switch.

Receive Side Scaling and ToE

If your multi-processor Exchange server is slamming one CPU(or core) and the rest are sitting idle, it’s a good bet you don’t have RSS enabled. RSS lets your server balance NIC interrupts across all the CPUs, which leads to better overall performance. It’s on by default in 2008 and 2008R2, but might have been turned off by another admin. If you see high CPU on only one processor, check with this command.

netsh interface tcp show global

If Receive-side Scalaing state shows as disabled, you’ve found the culprit.

That same command will also show you the status of TCP Chimney Offload, or ToE. With compatible NICs, ToE can provide much better throughput on large file transfers (like database replication for DAGs, mailbox moves, etc.) and reduced CPU utilization. With it off, those operations will take much longer, have lower throughput, and cause higher CPU utilization. 2008 disables ToE by default, while 2008 R2 uses an automatic setting. If your NICs support ToE, make sure you are using it by enabling it (if necessary) in the O/S, and then setting the advanced properties of the NIC to use it.

Using Hardware Load Balancers

The biggest challenge to troubleshooting load balanced servers is that the problem usually will manifest itself as intermittent, or isolated to a single client or subnet. If load balancers are in the mix, test from your machine, but test against the VIP and against each physical server one by one. If you cannot reproduce the problem, try the same process from the client. This may be one time where you have to use a HOSTS file to trick the client into connecting to each server one by one. If you don’t have admin access to the hardware load balancer, get on with that admin to do your tests so they can view realtime logs to see if anything stands out.

The Microsoft Network Load Balancing Service

If you are trying to load balance Exchange servers and are running into problems using software load balancing, my money is on the problem being in your switch configuration, and not with the MS NLB service. The easy test is to move the VIP to one of the servers, validate that everything works, and then move the VIP to the other and validate again. If it works without NLB in the mix, then it is not Exchange you should be looking at. MS NLB works great, though it is limited to IP based affinity and not port based, but there are so many ways the switch and/or router that your server connects to can screw up NLB, I’ll frequently recommend against using it unless I can directly manage the switches myself, or I know the person who does and that he or she understands their side of making NLB work.

See  http://technet.microsoft.com/en-us/library/ff625247.aspx for some more tips on MS NBL, and if you are using VMware to virtualize your servers, see this article for specific settings in VMware. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007371

Coming up next

In Part 5, we will look at the issues that can cause Exchange problems when making RPC calls, and how to troubleshoot those problems. Here’s a rundown of the six parts in this series. We’ll update with live links as each part is published over the next several weeks.

1. Introduction and DNS
2. Active Directory
3. Firewalls
4. NICs (this post)
5. RPCs
6. Client side issues

Troubleshooting Exchange Networking: NICs (Part 4)

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Often, Exchange administrators will receive escalated help desk tickets from users complaining that Exchange is “slow” and demanding resolution. These sorts of tickets (slow being at best a relative term and never specific enough about what precisely is considered to be slow) can be extremely challenging to work, since the subjective nature of slowness is often combined with an inability to replicate the problem, or the problem is intermittent. The Exchange admin can take a look at the server(s) for high CPU utilization, low memory conditions, disk and network queue lengths exceeding the norm, and finding nothing, shrug it back off to the desktop support team as a client issue. While it is often a client issue, there are several places between Outlook and a user’s mailbox that can cause intermittent slowness, and are fair to call networking bottlenecks. In a six part series of articles, we’ll look at how Exchange interacts on the network with various other services to help you identify network issues, and troubleshoot them when they occur.

In many cases, troubleshooting Exchange network bottlenecks will require a network trace, and may also require performance monitor counters. This series of articles will talk about both of those in general terms; how to use NetMon or Wireshark, and PerfMon are out of scope. In Part 3 of this series, we’re going to discuss the connectivity you need to permit through firewalls for Exchange to function properly on the network.

Firewalls

There are at least three places where a firewall can cause problems for Exchange. The most common is at your Internet border, when you are trying to support a protocol and the firewall is not permitting the necessary traffic. The second is between your DMZ and the internal network, which can cause issues for both Edge Transport servers and Client Access Servers, depending upon whether you pass traffic into them directly (which is not recommended) or you publish the CAS services using TMG or some other reverse web proxy. The third, which is both the least common and the most problematic, is when there are firewalls between different internal Exchange servers, or between Exchange servers and Active Directory.

Clients on the Internet must connect to the CAS servers for the various protocols they will use. Other Internet mail servers must connect to the Edge Transport server to exchange SMTP messages, and all Exchange server roles except the Edge Transport Server must query AD directly for configuration information, and to perform LDAP lookups for servers in different sites. They will also need to communicate with Active Directory to authenticate users. Edge Transport servers have to communicate with Hub Transport servers both to update their configuration, and to pass SMTP traffic in to the internal network. Any time a firewall is between two Exchange servers, or between an internal Exchange server and either Active Directory or any other part of the Exchange environment, you must ensure that all required traffic is permitted to pass through the firewall. Firewalls frequently translate IP addresses, called NAT. NAT is okay for some protocols; for others not so much. Windows 2008 and 2008 R2 servers will source all ephemeral connections from ports between 49152 and 65535. If you have any Exchange servers running 2003 or 2003 R2, you will need to expand that range to 1025-65535. The same can be said for clients. Windows Vista and 7 will source their connections from ports between 49152 and 65535. XP clients will source from 1025 to 65535.

Let’s look at each of the roles to see more about the required connectivity.

Edge Transport Server Role

Of course, your firewall needs to permit inbound TCP 25 from the Internet (ip any) to enable other Internet mail servers to send it email, and source ports can be anything from 1025 on up. You should also permit TCP port 587, which is commonly used by clients sending TCP over TLS connections. Older firewalls sometimes attempt to perform a rudimentary form of Intrusion Protection (fixup, inspect, etc.) which can often cause more problems than it solves, so consider carefully whether to enable that or not.

The Edge Transport server doesn’t access Active Directory directly, it stores it configuration in an instance of Active Directory Lightweight Directory Services. It uses an Edge Subscription to subscribe to a Hub Transport server in an Active Directory site, which will use the Microsoft Exchange EdgeSync service to synchronize Active Directory data to AD LDS. The Edge Transport server must be able to communicate to each and every Hub Transport server within the site it is subscribed to over TCP port 50636. That’s every Hub Transport server in the site, not just one or two, and it will source its queries from an ephemeral port between 49152 and 65535. If you add a Hub Transport server to the site, you must update your firewall rules to include the new server and update your Edge subscription.

You can use NAT for both Internet traffic in to the Edge Transport server, and from the Edge Transport server into the Hub Transport servers in the subscribed site.

Hub Transport Server Role

The Hub Transport server must contact Active Directory to perform message categorization, necessary for recipient lookup and routing resolution. This will include the location of the recipient’s mailbox and any restrictions or permissions that may apply. It will also use LDAP queries to expand the membership of distribution lists to determine membership of a dynamic distribution list.

It’s best if there is no firewall between a Hub Transport server and the Domain Controllers in the same site, but if you must place a firewall between them, ensure that the Exchange server can reach all Domain Controllers in the site over all the following ports and protocols.Collapse this tableExpand this table

Collapse this tableExpand this table

NAT is no good here; it can break RPC DCOM traffic which is used for some Active Directory functions.

Client Access Server Role

The Client Access server role services clients connecting from the Internet who want to use Outlook Web App, POP3, IMAP4, or ActiveSync. When a connection is received, the Client Access server authenticates the user against AD and then queries to determine the appropriate mailbox server. If the user’s mailbox is in the same site, the user is connected directly to their mailbox. If in a different site, the connection is redirected to a Client Access server in the remote site.

If you are going to provide client connections directly to the CAS server, you must permit the following for the relevant client protocols.Collapse this tableExpand this table

Unified Messaging Server Role

The Unified Messaging server will need essentially the same connectivity as the Hub Transport server role, plus whatever required ports are necessary for your particular VoIP gateway. Consult your vendor’s documentation for those specifics.

Mailbox Server Role

The Mailbox server will also need the same connectivity as detailed for the Hub Transport server role.

Limiting RPC ports

Firewall admins don’t like to carve large holes in their walls, and will often request that you limit the port ranges used by RPC connections. This is supported, and well documented, but be warned. It is very common to limit RPC connections to too narrow a range of ports. This will manifest as random failures particularly at peak load times, with tons of 1722 errors. If you must restrict RPC ports, I suggest you start with a range of at least 1000 ports, and carefully monitor clients and servers to ensure that this is enough to support all connections during peak times.

Troubleshooting Exchange firewall issues

Knowing the ports Exchange uses will help you troubleshoot issues. If you suspect Exchange is having a problem caused by a firewall, it’s best if you can work directly with the firewall administrator, who can monitor the source and/or destination IP addresses to see if rules are blocking. If that is not possible, you can test connectivity between Exchange and Active Directory or other Exchange servers by using the PortQueryUI tool. You can also use PING, the TCPING tool, or even the Windows Telnet client to see whether you can connect to the port or not.

PortQueryUI can provide specific success or failures, but you can use PING to make sure you can reach the destination server, and then TCPING or Telnet to confirm whether or not you can make a connection on the specific ports required. If you get timeouts or refusals, and you have confirmed the destination server is up and running, then you are probably dealing with a firewall issue. There’s no real workaround here; the firewall admin must permit the required traffic for all services.

Coming up next

In Part 4, we will look at the issues that can cause Exchange problems when NICs are involved, and how to troubleshoot those problems. Here’s a rundown of the six parts in this series. We’ll update with live links as each part is published over the next several weeks:

1. Introduction and DNS
2. Active Directory
3. Firewalls (this post)
4. NICs
5. RPCs
6. Client side issues

Troubleshooting Exchange Networking: Firewalls (Part 3)

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Often Exchange administrators will receive escalated help desk tickets from users complaining that Exchange is “slow” and demanding resolution. These sorts of tickets (slow being at best a relative term, and never specific enough about what precisely is considered to be slow) can be extremely challenging to work, since the subjective nature of slowness is often combined with an inability to replicate the problem, or the problem is intermittent. The Exchange admin can take a look at the server(s) for high CPU utilization, low memory conditions, disk and network queue lengths exceeding the norm, and finding nothing, shrug it back off to the desktop support team as a client issue. While it is often a client issue, there are several places between Outlook and a user’s mailbox that can cause intermittent slowness, and are fair to call networking bottlenecks. In a six-part series of articles, we’ll look at how Exchange interacts on the network with various other services to help you identify network issues, and troubleshoot them when they occur.

In many cases, troubleshooting Exchange network bottlenecks will require a network trace, and may also require performance monitor counters. This series of articles will talk about both of those in general terms; how to use NetMon or Wireshark, and PerfMon are out of scope. In Part 2 of this series, we’re going to discuss how Exchange is dependent upon and interacts with Active Directory on the network.

Active Directory

There’s a ton of network interactions between Exchange servers and Active Directory, which is why you are required to have a Global Catalog server in every site in which you have an Exchange server. An Active Directory site is usually defined as a collection of subnets with sufficient bandwidth to support replication, and that can lead to sites spanning WAN links. While the WAN may have sufficient bandwidth and low enough latency to support Active Directory replication and authentication traffic, any AD client that is in a site may connect to, and query, and Domain Controller within that site. When the target of queries is across the WAN, the total latency of the WAN link can add up to noticeable delays. Understanding just how much goes on between your Exchange server and your Global Catalog server may be enough to make you change the word “site” to “subnet.” Exchange servers will bind to a randomly selected domain controller and global catalog server in the same site, to minimize WAN traffic. Ensure that there are redundant servers will keep WAN traffic to a minimum, and optimize Exchange performance.

Note: Read-Only domain controllers are not usable by Exchange. Exchange must access writable domain controllers.

Configuration information

The configuration partition in Active Directory contains critical data about the forest-wide configuration. Exchange configuration information can be found in a subfolder of the Services container in the Configuration partition. This includes:

1. Address lists
2. Address and display templates
3. Administrative groups
4. Client access settings
5. Connections
6. Messaging records management, mobile, and UM mailbox policies
7. Global settings
8. E-mail address policies
9. System policies
10. Transport settings

All Exchange server roles, except the Edge Transport Server, will query AD directly for this information. Here’s more specific information on how each role depends upon AD. You can also read more about that here http://technet.microsoft.com/en-us/library/aa998561.aspx.

Hub Transport Server Role

The Hub Transport server must contact Active Directory to perform message categorization, necessary for recipient lookup and routing resolution. This will include the location of the recipient’s mailbox and any restrictions or permissions that may apply. It will also use LDAP queries to expand the membership of distribution lists to determine membership of a dynamic distribution list.

The Hub Transport Server will use cached information regarding the AD site topology to determine routing for message delivery between sites. If the Hub Transport server determines that a mailbox is in the same site, it will deliver the message directly to the Mailbox server, otherwise it will route the message to a Hub Transport server in the destination site.

The Hub Transport server uses the application partition of Active Directory to store and access configuration information, including transport rules, journal rules, and connectors.

Client Access Server Role

The Client Access server role services clients connecting from the Internet who want to use Outlook Web App, POP3, IMAP4, or ActiveSync. When a connection is received, the Client Access server authenticates the user against AD and then queries to determine the appropriate mailbox server. If the user’s mailbox is in the same site, the user is connected directly to their mailbox. If in a different site, the connection is redirected to a Client Access server in the remote site.

Unified Messaging Server Role

The Unified Messaging server queries Active Directory to retrieve global configuration information, such as dial plans, IP gateways, and hunt groups. When a message is received by the Unified Messaging server, it matches the telephone number to a recipient address, then the location of the user’s mailbox. It can then route the voicemail message to a Hub Transport server for delivery to the mailbox.

Mailbox Server Role

The Mailbox server also stores configuration information Active Directory, including agent configuration, address lists, and policies. The Mailbox server will use this to enforce mailbox policies and global settings.

Edge Transport Server Role

The Edge Transport server doesn’t access Active Directory. It stores it configuration in an instance of Active Directory Lightweight Directory Services. It uses an Edge Subscription to subscribe to a Hub Transport server in an Active Directory site, which will use the Microsoft Exchange EdgeSync service to synchronize Active Directory data to AD LDS.

Site definitions

There are two rules of thumb for Active Directory site design and how it impacts Exchange:

1. Make sure every single subnet that hosts an Exchange server belongs to a site
2. Don’t let any of those sites span the WAN, no matter how much bandwidth you have available.

If an Exchange server cannot determine its AD site because the subnet does not belong to a site, the MSExchangeDSA will fail with a 2114 and MSExchangeSA will fail with a 1005. In both cases it is because Exchange could not determine the AD site based on the subnet. Even the fastest WAN links have higher latency than the slowest LAN links, and that latency will have a cumulative and negative impact on Exchange performance as the server is waiting on responses from domain controllers if the DC is on the far side of the WAN from the Exchange server.

Troubleshooting Exchange interaction with Active Directory

Knowing how Exchange depends upon Active Directory will help you troubleshoot issues. The four main categories of problem are:

1. Network latency between the Exchange server and GC/DC
2. Firewall rules blocking connection attempts
3. Incorrect site configuration
4. Replication problems within AD

If you suspect Exchange is having a problem accessing Active Directory, first ensure that Exchange can communicate with a domain controller for each domain in the forest that has users with mailboxes, and that there is at least one domain controllers in the same site that is a global catalog server. Look for errors including 2114, 1005, and 1722.

Test connectivity between Exchange and Active Directory by using the PortQueryUI tool, and the response times to LDAP queries using LDP.EXE and a protocol analyzer. And of course, ensure that you have no replication problems with your Active Directory. A domain controller that stops replicating because of DNS islanding or other connectivity issues with the rest of the forest will directly impact AD. Changes in AD (like name, group membership, SMTP proxy addresses, etc.) must replicate to all domain controllers that Exchange relies upon before you can be sure that Exchange will pick up on/display the differences.

Performance will be enhanced by redundancy. When possible, ensure that there are multiple global catalog servers in the same site as every Exchange server, and that every domain in the forest with Exchange users is represented.

Performance of Exchange will also improve directly with the capabilities of those domain controllers. When the DC is able to cache the entire Active Directory in memory, response to queries from Exchange will be much faster. Look at implementing 64bit DCs with enough RAM to cache the entire database.

On a domain controller a quick way to check for replication problems is to run this command in an administrative command prompt

Repadmin /replsummary [enter]

Check for fails, servers that are down or unreachable, and larger times since the last replication event.

Coming up next

In Part 3, we will look at the connectivity requirements for Exchange as they relate to firewalls, and how to troubleshoot those problems. Here’s a rundown of the six parts in this series. We’ll update with live links as each part is published over the next several weeks:

1. Introduction and DNS
2. Active Directory
3. Firewalls
4. NICs
5. RPCs
6. Client side issues

Troubleshooting Exchange Networking: Active Directory (Part 2)

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Whether you are troubleshooting an Exchange server performance issue, trying to see how well you sized your servers, or just want a better idea of what your users are doing, the Exchange Server User Monitor from Microsoft (or ExMon as it is known to its friends) is a great, free tool you can use to gather all sorts of information about your Exchange environment. The Exchange Server User Monitor has been around for years, and this latest version, 14.2.247.5, was released in December of 2011.

You can download ExMon from this link and use to evaluate a server, or an individual user’s interactions with that server. As with many tools from Microsoft, this has been around for years, but gets an update and a facelift every so often. With ExMon, you can view the following information:

• IP addresses used by clients
• Microsoft Office Outlook® versions and mode, such as Cached Exchange Mode and classic online mode
• Outlook client-side monitoring data
• Resource use, such as:
  • CPU usage
  • Server-side processor latency
  • Total latency for network and processing with Outlook 2003 and later versions of MAPI
  • Network bytes
  • And more.

The download is a simple MSI file that weighs in under 2MB in size, and the install is of the next agree next enter variety. You don’t need to run this tool on your Exchange server; It can run just fine on another server or on your workstation when you want to use it to view trace files gathered by the tool running on an actual Exchange server. Just launch it from the command line passing the ETL filename in the command, like exmon.exe c:\temp\exch01.etl [enter]. Note, if you are going to run the tool on your workstation, you can find it at C:\Program Files (x86)\Exchange User Monitor. There’s a reg file in that directory that you should import into your registry so the tool can work properly.

You can collect data for use with ExMon in one of three ways:

• Collecting data directly with ExMon
• Collecting data by using System Monitor (Windows 2000 Server and Windows Server 2003 only)
• Collecting data by using command-line tools.

Using ExMon directly to collect data is best done when you are looking to “spot check” a server and plan to gather data for only short intervals. ExMon trace files can become very large, especially when the monitor interval is long, and parsing these files can be both CPU and RAM intensive.

For trending data, it’s best to use System Monitor, and schedule it with a reasonable sampling frequency. It’s best to start out small, monitor the size of the files generated, and adjust your sampling interval and the duration of your monitoring as you see fit.

While the documentation has not been updated yet for this version, you can read more about how to use ExMon at the TechNet site: http://technet.microsoft.com/en-us/library/bb508855(EXCHG.65).aspx.

Cool Tools: Microsoft Exchange Server User Monitor

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Often Exchange administrators will receive escalated help desk tickets from users complaining that Exchange is “slow” and demanding resolution. These sorts of tickets (slow being at best a relative term, and never specific enough about what precisely is considered to be slow) can be extremely challenging to work, since the subjective nature of slowness is often combined with an inability to replicate the problem, or the problem is intermittent. The Exchange admin can take a look at the server(s) for high CPU utilization, low memory conditions, disk and network queue lengths exceeding the norm, and finding nothing, shrug it back off to the desktop support team as a client issue. While it is often a client issue, there are several places between Outlook and a user’s mailbox that can cause intermittent slowness, and are fair to call networking bottlenecks. In a six part series of articles, we’ll look at how Exchange interacts on the network with various other services to help you identify network issues, and troubleshoot them when they occur.

In many cases, troubleshooting Exchange network bottlenecks will require a network trace, and may also require performance monitor counters. This series of articles will talk about both of those in general terms; how to use NetMon or Wireshark, and PerfMon are out of scope. In Part 1 of this series, we’re going to discuss how Exchange is dependent upon and interacts with DNS on the network.

DNS

DNS is one of the most important, and fundamental services on any TCP/IP network and the critical role it plays in all aspects of Exchange cannot be understated. Every single interaction between servers depends on being able to resolve a name to an IP address, and being able to quickly (and correctly) perform name resolution can set the tone for the entire transaction.

Most of you will be using AD integrated DNS, so your DNS servers will be domain controllers. Keep in mind that the default TTL for AD integrated zones is 3600, so your Exchange servers will cache responses for an hour before trying to resolve the same name again. Using AD integrated zones also means that changes to DNS records must replicate to all domain controllers, and then the TTL must expire before you can assume that a client or Exchange server is resolving the right IP address to name.

To ensure that the right IP address is being provided in response to a query, open an administrative command prompt on the Exchange server you are troubleshooting, and use the NSLOOKUP command to query the primary DNS server, and the secondary. Confirm that both provide the same result and that it is correct, and then ping the destination server by name. Compare the IP address in the PING command to what NSLOOKUP returned to be sure that your Exchange server is trying to reach the right address. If it is not, issue the ipconfig /flushdns command to clear the local cache, and try again.

>nslookup exch2.example.com
Server:  dc1.example.com
Address:  192.168.0.2
Name:    exch2.example.com
Address:  192.168.0.6
>ping exch2.example.com
Pinging exch2.example.com [192.168.0.9] with 32 bytes of data:
Reply from 192.168.0.104: Destination host unreachable.
Ping statistics for 192.168.0.9:
    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),

>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.

>ping exch2.example.com
Pinging exch2.example.com [192.168.0.6] with 32 bytes of data:
Reply from 192.168.0.6: bytes=32 time=4ms TTL=128
Ping statistics for 192.168.0.6:
    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 2ms, Average = 2ms

You want to place DNS servers as “close” to your Exchange servers as possible, configure your Exchange servers to use the closest DNS servers they can, and to keep the application response time (ART) for DNS queries as low as possible. If it takes more than 50 milliseconds to resolve a DNS performance will suffer. You can use a protocol analyzer like Microsoft’s NetMon or Wireshark to analyze that, or you can just use the dig command. A Windows port can be downloaded from here. The dig command can tell you how long it takes to resolve a name.

>dig @192.168.0.2 -t a exch2.example.com
; <<>> DiG 9.3.2 <<>> @192.168.0.2 -t a exch2.example.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 104
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;exch2.example.com.             IN      A

;; ANSWER SECTION:
exch2.example.com.      3600    IN      A       192.168.0.6
;; Query time: 8 msec
;; SERVER: 192.168.0.2#53(192.168.0.2)
;; WHEN: Fri Dec 30 15:29:26 2011
;; MSG SIZE  rcvd: 51

Eight milliseconds is not bad at all.

Your internal Exchange servers (CAS, HUB, UC, and Mailbox) should be configured to use local servers for both their primary and secondary DNS. In sites where there is only DNS server, you really ought to add another, but if you cannot, configure the secondary to be the one with the least latency. That won’t always be the one on the other side of the connection with the greatest bandwidth; test.

Your Edge Transport servers should be configured to resolve DNS queries to servers as close to the Internet edge as possible, and these should be able to go straight to root rather than forwarding to your ISP. That way, every MX lookup, SPF lookup, DKIM lookup, and PTR lookup that the Edge must perform when sending or receiving a message can complete as quickly as possible. Configuring the Exchange server to query an internal DNS server, which then must forward to your ISP, which then may forward to another, adds lots of latency to every DNS lookup. Sure, the operating system will cache those lookups, but caches expire and you are exchanging email with hundreds or thousands of domains each day. Keep in mind that changes beyond your control will be made as other admins move their services to different servers, networks, etc. Changes to DNS records take time to replicate; if you are troubleshooting a connectivity failure to a remote system, don’t forget that they may be in the middle of a change and DNS records are simply stale. Time will sort that out for you.

Considering that DNS queries must be resolved in order for an Exchange server to connect to the Global Catalog server, which it must do for authentication, to expand distribution lists, to look up topology information, and to do practically anything else, and you will understand that you don’t want to waste time just trying to resolve a name to an IP address.

Coming up next

In Part 2, we will look at how Exchange interacts with Active Directory at the network level, where bottlenecks can occur, and how to troubleshoot those problems. Here’s a rundown of the six parts in this series. We’ll update with live links as each part is published over the next several weeks.

1. Introduction and DNS
2. Active Directory
3. Firewalls
4. NICs
5. RPCs
6. Client side issues

Troubleshooting Exchange Networking: DNS (Part 1)

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Users interested in deploying a hybrid configuration have been looking forward to Exchange 2010 SP2 for months so they could take advantage of the new hybrid configuration wizard included with SP2. That wizard takes dozens of manual steps and automates them in a simple to follow wizard, which we discussed in this article a few weeks ago.

While the hybrid configuration wizard is a great improvement in setting up an Exchange system with some mailboxes on premise, and others with a cloud service provider, it seems a small glitch made it through to the release of SP2. It seems that many customers are running into issues using PKI certificates that were previously issued and which worked without a problem in Exchange 2010 RTM and/or SP1.

There is a TechNet article called Understanding Certificate Requirements for Hybrid Deployments that details what you should do when creating a certificate for hybrid deployments. The article (as of the date this post was written) only indicates SP1, but includes the steps I followed when creating a certificate for hybrid configuration. The article discusses the use of a SAN certificate, but does not discuss using a wildcard certificate, and here’s why this is a good thing. When you run the wizard to set up hybrid configuration, the wizard parses the CN of your certificate and attempts to set up a Send Connector for the SMTP encryption between your on-premise and your remote Exchange infrastructure. If it encounters a value like *.example.com in the CN, the wizard will error out because that is an invalid name for a Send Connector. Here’s what the error looks like:

Update-HybridConfiguration

Failed

Error:

Updating hybrid configuration failed with error
'Subtask Configure execution failed: Configure Mail Flow
Execution of the New-SendConnector cmdlet had thrown an exception.
This may indicate invalid parameters in your Hybrid Configuration settings.
Cannot process argument transformation on parameter 'Fqdn'.
Cannot convert value "*.example.com" to type "Microsoft.Exchange.Data.Fqdn".
Error: ""*.example.com" isn't a valid SMTP domain."
at System.Management.Automation.PowerShell.CoreInvoke[TOutput]
(IEnumerable input, PSDataCollection`1 output, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.Invoke(IEnumerable input,
PSInvocationSettings settings) at System.Management.Automation.PowerShell.Invoke()
at Microsoft.Exchange.Management.Hybrid.RemotePowershellSession.RunCommand
(String cmdlet, Dictionary`2 parameters, Boolean ignoreNotFoundErrors)'.
Additional troubleshooting information is available in the Update-HybridConfiguration
log file located at C:\Program Files\Microsoft\Exchange
Server\V14\Logging\Update-HybridConfiguration\HybridConfiguration_12_16_2011_5_58_59_634596119396658235.log.
Exchange Management Shell command attempted:
Update-HybridConfiguration -OnPremisesCredentials
'System.Management.Automation.PSCredential' -TenantCredentials 'System.Management.Automation.PSCredential'

So, what can you do to move past this? Two choices are available. The first is to not use a wildcard certificate. I know, I know, wildcard certs are awesome, solve a ton of other headaches, and security concerns notwithstanding, are a dream come true. However, since the * in the wildcard cert is what causes the wizard to hurl, stick with a SAN certificate if you need a cert that can validate more than one name. The second is to get the fix from Microsoft. If you already have a wildcard certificate, this is the more economical way to go. You can wait for RU1 that is due to release in January 2012, or you can contact Microsoft for a hotfix.

Even with this little “issue” SP2 is a great improvement over SP1, and if you are planning a hybrid deployment, it is still the way to go. A regular, SAN, or UC certificate is far less expensive than a wildcard, so this may not be an issue for you anyway, but if you already have a wildcard cert, your fix is a free phone call away.

Certificate Problems with Hybrid Configuration in SP2

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Now that Exchange 2010 SP2 is available for download, I’m sure many of you (like me) have already downloaded the binary and are testing it in the lab. Of course, the reason we test is because we want to ensure that we don’t create problems in production which is prudent and a best practice for administration. SP2 is a great service pack, and in a vanilla Exchange 2010 organization I seriously doubt you will encounter a single issue with this service pack, but how many of us are running a vanilla org, freshly installed from scratch? For the majority of us who aren’t, here are some pointers about SP2 that should prove useful.

Network timeouts and long installation times

The Exchange 2010 SP2 binary is a slipstreamed copy of Exchange 2010 with SP2. You can use it to patch an existing server, or to install a new server from scratch, so keep it handy, but also keep in mind it isn’t exactly small. The download is 535 MB, but when you run it, it will expand to 1.38 GB. Make sure you have room for that wherever you decide to expand it, and consider whether to place it on a network share where all your Exchange servers can access it, or if you should copy the downloaded EXE to Exchange servers in remote offices before you expand it.

Schema extension errors

Yes, you must extend the schema for SP2. That means you need Schema Admin rights, or to have your AD administrator extend the schema before you can apply SP2 to any server. If you are not also the AD admin, engage that person now.

CAS Server update fails

SP2 requires some additional components for CAS servers that SP1 and RTM did not. Make sure that your CAS server has the following IIS role services installed before applying SP2, or it will fail. If you are running Windows 2008 SP2 use Server Manager to install:

• IIS 6 WMI Compatibility
• ASP.NET
• ISAPI Filters
• Client Certificate Mapping Authentication
• Directory Browsing
• HTTP Errors
• HTTP Logging
• HTTP Redirection
• Tracing
• Request Monitor
• Static Content

If you are running Windows 2008 R2 you can use PowerShell to install the required modules by running:

Import-Module ServerManager [enter]

Add-WindowsFeature Web-WMI,Web-Asp-Net,Web-ISAPI-Filter,Web-Client-Auth,Web-Dir-Browsing,
Web-Http-Errors,Web-Http-Logging,Web-Http-Redirect,Web-Http-Tracing,Web-Request-Monitor,
Web-Static-Content [enter]

If that’t too much effort, you can install SP2 in unattended mode like this in a normal administrative command prompt.

Setup /Mode:Upgrade /InstallWindowsComponents [enter]

Errors managing RBAC

SP2 changes some of the Role Based Access Control definitions in Active Directory. If you try to manage any RBAC roles from a server that has not yet been updated, you will encounter errors in both the Exchange Management Shell, and the Exchange Control Panel.

In the shell you will see:
WARNING: The object MyMailboxDelegation has been corrupted, and it’s in an inconsistent state. The following validation errors happened:
WARNING: The property value you specified, “15″, isn’t defined in the Enum type “ScopeType”.

In the control panel you will see:
There are multiple warnings. Click here to see more
The object MyMailboxDelegation has been corrupted, and it’s in an inconsistent state. The following validation errors happened:
The property value you specified, “15″, isn’t defined in the Enum type “ScopeType”.

Upgrade all Exchange servers to SP2, or use a server that has already been upgraded to manage RBAC until you can finish patching the other servers.

Redirs for OWA fail

If you are using a simple URL and not requiring HTTPS (like http://mail.example.com) to redirect your users to their OWA, this will fail after updating to SP2. To avoid this, as soon as SP2 has been applied to the CAS server, modify your web.config file using the steps found in http://technet.microsoft.com/en-us/library/aa998359.aspx.

Cross-forest mailbox moves fail

If you have a multi-forest Exchange org, or are migrating to Office 365, this is a big one. The way MRSProxy works has changed with SP2, so the service is disabled by SP2 and settings in the EWS\web.config file are no longer used. Use the EMS command to reenable the MRSProxy.

Set-WebServicesVirtualDirectory -Identity "EWS (Default Web Site)" -MRSProxyEnabled $true [enter]

Hybrid Configuration Wizard fails

There’s a known issue setting up hybrid configuration using the wizard if the FQDN of your Hub Transport server starts with a number. You can either use a different HT server, rename your HT server, or use the EMS Update-HybridConfiguration cmdlet to set up hybrid coexistence instead of using the wizard.

Knowing these ahead of time can help to ensure your testing, and production deployment, of SP2 goes off without a hitch. Good hunting!

Troubleshooting Exchange 2010 SP2 Installation

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

You’ve probably seen this before. A user opens a help desk ticket because every time they try to send an email to someone, it bounces. That someone could be a co-worker using the same email system, or it could be a customer on an external email system – it doesn’t matter. When the user replies to an email sent from the other person, the reply is delivered flawlessly. But when the user tries to create a new email, it bounces.

You try to send an email to the remote person and it is delivered correctly. You use message tracking to try to run down the problem with the email, and you might not even find it (if you are searching on the recipient address that is…which is a hint).  It’s not until you have the users actually showing you what they are doing that you realize they have a bad address in their nickname cache.

The nickname cache, which provides Outlook’s handy auto-completion when you start to type a name or an email address into the TO: or CC: or BCC: boxes in a new email, is used both to perform automatic name checking and to perform auto-completion. It is also lets you start to type “Cas…” into the TO: box and pulls up casper.manes@example.com so you don’t have to type out the complete email address. The problem comes up when a recipient’s address is wrong, or changes, and your client holds old or bad information.

To fix this, you can remove entries one at a time, or you can purge the cache completely. If you have recently changed your internal addressing standard, or migrated to a new system, I tend to just purge the whole thing so folks have to go to the GAL for fresh information. They will rebuild their cache soon enough, but if it is just a one or two addressee issue, removing individual entrees is easy enough.

To remove a single entry:

1. Start typing the email address, until autocomplete provides choices, like shown below.
   
2. Click the X to the right of the name to delete it from the cache.

To completely remove the cache:

1. Click File, Options
2. Select the Mail tab
3. Scroll down to “Send messages” and click the “Empty Auto-Complete List” button.
   

Alternatively, you can launch Outlook from the Run dialog using
Outlook.exe /CleanAutoCompleteCache

Protip: using that cmd line in a login script is a convenient way to clear all users’ caches after a migration.

Removing bad entries will force the user to go to the GAL, or use a personal contact, or just type the email address in longhand, which will update the nickname cache with the proper email address, and that means a problem solved.

Troubleshooting Outlook Auto-complete

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

The Offline Address Book is one of those services offered by Exchange that usually just works. It enables disconnected users, or those running Outlook in cached mode, to maintain a copy of the Global Address Book and/or other address books on their local machine so that they can look up users on the go. And when it works, it works very well, but sometimes even the best services go awry and require some troubleshooting. And when that’s the OAB, knowing how to identify and resolve the problem gets those remote users back up and running in no time.

There are several things that can cause problems with the OAB. We’ll go over these based on whether you want to look at the server, the client, or the network, and we’ll see what you can do to fix any problems you find.

Server side

Unless you have the client standing in your cubicle pointing at their laptop and saying “make better,” you will probably start your efforts on the mailbox server. Here are the things that can most often cause the OAB problems:

Disk space

The OAB creation and update processes generate several temp files in %TEMP% and store the actual OAB files under c:\Program Files\Microsoft\Exchange Server\V14\ExchangeOAB. If you built your VM with too small a C: drive, you can of course change the %TEMP% variable to another drive using the System Properties, and move then update the OAB temp file location using this reg key.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\MSExchangeSystemAtt
endant\OALGEN_REG_V4_ALT_TEMP_FILES_LOCATION

Note: this only works for v4 type OABs. If you are supporting legacy clients, upgrade, or find a way to free up more space on C:.

Antivirus

Antivirus real-time scanning can cause all kinds of problems, and common problem with a/v and OAB is indicated by Event IDs 9373, 9109, and 9116 long after an initial OAB generation. Now I will never advocate disabling a/v, but you do want to make sure you have proper exceptions set up for Exchange directories. See Dave Goldman’s blog on MSDN, http://blogs.msdn.com/b/dgoldman/archive/2010/05/12/proper-virus-exclusions-for-servers-hosting-the-oab.aspx for the specific exclusions.

Timing

OAB updates run once a day; at 05:00 local time by default. You can update Active Directory, force replication between DCs and confirm it, and then have your clients download the OAB again, but if you don’t remember to force the OAB update process to run, they still won’t pick up the changes. OAB generation can be CPU intensive, so I prefer to set the expectation with HR and users that they need to provide changes a day in advance, but when that just won’t do, ensure that AD replication has completed, and then run this cmdlet on the Mailbox server in the EMS;

Update-OfflineAddressBook –Identity “Default Offline Address Book”

Permissions

Sometimes a well-meaning but misguided security guideline might recommend permissions changes to servers as part of a “hardening guide”. While many of these are very useful, if they are not written specifically for Exchange 2010, and role specific, then in the words of Gandalf the Grey, “run you fools!” The default permissions for the OAB directory are as follows:

• Allow SYSTEM Full Control
• Allow Administrators: Full Control
• Allow Exchange Servers:
• Traverse Folder
• List Folder
• Read Attributes
• Read Extended Attributes
• Read Permissions
• CONTAINER_INHERIT (folder and subfolders permissions)
• Allow Exchange Servers:
• Read Data
• Read Attributes
• Read Extended Attributes
• Read Permissions
• CONTAINER_INHERIT + OBJECT_INHERIT (folder, subfolders and files permissions)

If you find it easier, you can delete the OAB directory structure c:\Program Files\Microsoft\Exchange Server\V14\ExchangeOAB and the OABgeneration process will recreate it with the correct perms.

Active Directory replication

Never overlook the possibility that AD replication issues are causing Exchange issues. The OAB pulls its updates from a Global Catalog server in the same site, and if the GC it queries is not synchronizing with the rest of AD, you will need to fix that; then your OAB will be fine.

Client side

Client side problems may require that you get a remote session going with your user, since a picture is worth a thousand words, but you don’t want to use those thousand words trying to talk a user through some of these:

Distribution (Outlook 2010/2007/2003)

Outlook 2010 and 2007 are the only versions still supported, but we all know there is a ton of Outlook 2003 still out there. Exchange 2010 defaultly will only distribute the OAB using a virtual directory on the CAS server. If you have older clients, you will need to create a Public Folder database and set the OAB to also distribute through Public Folders.

Antivirus (again)

Misconfigured a/v software on the client can cause it to try to download a complete copy of the OAB again and again. Exempt *.oab files on your client from real-time scanning.

On the network

Bandwidth

By default, there is no limit to the number of concurrent downloads of the OAB, nor is there any limit to the amount of traffic this can consume. If many clients attempt to pull down the OAB at the same time, it could overload the network connection, the CAS server, or the Mailbox server hosting the Public Folder containing the OAB (assuming you have legacy clients to support.) While this issue should pass on its own eventually, as once all clients have the OAB they will only download deltas, if you are supporting a network with thousands of users all trying to get the full OAB at once, you may run into problems. See this MSDN blog post, also from Dan Goldman, on steps to take to throttle the bandwidth. http://blogs.msdn.com/b/dgoldman/archive/2009/10/21/how-can-we-try-to-control-the-network-consumption-by-using-oab-throttling.aspx

CAS server problems

The mailbox server creates the OAB, but it’s the CAS server that distributes it to users, picking the OAB up from the mailbox server. If the File Distribution Service (FDS) is not running on the CAS server, or if there is a firewall between the CAS and Mailbox server blocking RPC and CIFS, the CAS won’t be able to pick up the OAB. Check the system log for problems with FDS, and if you must have a firewall between the CAS and Mailbox servers, make sure RPC (TCP 135, 49152-65535) and CIFS (TCP 445) are open. You also want to make sure that the CAS server has the proper a/v exclusions set up, as mentioned above.

The OABInteg Utility

The OABInteg utility can be used to simulate a client downloading the OAB from a server, and also to check the OAB for errors. You can download the OABInteg tool from http://code.msdn.microsoft.com/oabinteg/Release/ProjectReleases.aspx?ReleaseId=726
and read about how to use it at http://support.microsoft.com/kb/907792.

Now you have an idea of what can cause OAB problems, how to troubleshoot many of the issues, and a tool you can use to dig deeper into the OAB if necessary. Good hunting!

Troubleshooting the Offline Address Book

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Even the best run Exchange implementation will require troubleshooting from time to time, and in many cases, that troubleshooting will involve client access. Client access can be particularly challenging when the client is not in the cube next to you.  Questions like “what are the symptoms? Has anything changed on your workstation? What error message did you get?” tend to overwhelm many non-technical users, and sometimes it’s just not convenient to walk over to the user’s desk, especially when they work in another timezone.

Figuring out the scope of the problem is crucial, and then you need to determine whether the problem is on the server side or the client side of the equation. When you cannot ‘touch’ the client, how can you test the client’s side of things?

With several of the cmdlets included in the Exchange Management Shell, that’s how. The EMS includes so many commands, it’s almost impossible to keep them all straight, even if you use the shell exclusively and never touch your mouse. There is so much great information on Microsoft’s TechNet site, but finding what is useful can be challenging when you are trying to troubleshoot an urgent issue.

That’s where this article comes to your aid. It lists some of the most useful cmdlets for troubleshooting client access, includes the command, the description, and the most useful example from the relevant TechNet articles, and links to the article for your reference. Bookmark this post as a reference, and use its links to refer to the TechNet article whenever a client issue comes up. Soon you will have them committed to memory.

Test-OutlookConnectivity

Use the Test-OutlookConnectivity cmdlet to test end-to-end Microsoft Outlook client connectivity in the Microsoft Exchange Server 2010 organization. This includes testing for Outlook Anywhere (RPC/HTTP) and TCP-based connections. This example tests the most common end-to-end Outlook connectivity scenario for Outlook Anywhere. This includes testing for connectivity through the Autodiscover service, creating a user profile, and logging on to the user mailbox. All of the required values are retrieved from the Autodiscover service. Because the Identity parameter isn’t specified, the command uses the temporary test user that you’ve created using the New-TestCasConnectivityUser.ps1 script. This example command can be run to test TCP/IP connectivity by setting the Protocol parameter to RPC.

Test-OutlookConnectivity -Protocol:HTTP -GetDefaultsFrom
AutoDiscover:$true

Test-ActiveSyncConnectivity

Use the Test-ActiveSyncConnectivity cmdlet to perform a full synchronization against a specified mailbox to test the configuration of Microsoft Exchange ActiveSync. This example tests the Exchange ActiveSync connectivity for the mailbox PaulS using the Autodiscover URL.

Test-ActiveSyncConnectivity -UseAutodiscoverForClientAcces
sServer $true -URL "http://contoso.com/mail" -MailboxCrede
ntial "pauls@contoso.com"

Test-MapiConnectivity

Use the Test-MapiConnectivity cmdlet to verify server functionality by logging on to the mailbox that you specify. If you don’t specify a mailbox, the cmdlet logs on to the SystemMailbox on the database that you specify. This example tests connectivity to a mailbox, specified as a domain name and user name.

Test-MapiConnectivity -Identity "midwest\john"

Test-OutlookConnectivity

Use the Test-OutlookConnectivity cmdlet to test end-to-end Microsoft Outlook client connectivity in the Microsoft Exchange Server 2010 organization. This includes testing for Outlook Anywhere (RPC/HTTP) and TCP-based connections. This example tests the most common end-to-end Outlook connectivity scenario for Outlook Anywhere. This includes testing for connectivity through the Autodiscover service, creating a user profile, and logging on to the user mailbox. All of the required values are retrieved from the Autodiscover service. Because the Identity parameter isn’t specified, the command uses the temporary test user that you’ve created using the New-TestCasConnectivityUser.ps1 script. This example command can be run to test TCP/IP connectivity by setting the Protocol parameter to RPC.

Test-OutlookConnectivity -Protocol:HTTP -GetDefaultsFromAu
toDiscover:$true

Test-OutllookWebServices

Use the Test-OutlookWebServices cmdlet to verify the Autodiscover service settings for Microsoft Outlook on a computer running Microsoft Exchange Server 2010 that has the Client Access server role installed. This example tests for a connection to each service. This example also submits a request to the Availability service for the user holly@contoso.com to determine whether the user’s free/busy information is being returned correctly from the Client Access server to the Outlook client.

Test-OutlookWebServices -Identity:holly@contoso.com

Test-OwaConnectivity

Use the Test-OwaConnectivity cmdlet to verify that Microsoft Office Outlook Web App is running as expected. The Test-OwaConnectivity cmdlet can be used to test Outlook Web App connectivity for all Microsoft Exchange Server 2010 virtual directories on a specified Client Access server for all mailboxes on servers running Exchange that are in the same Active Directory site. The Test-OwaConnectivity cmdlet can also be used to test the connectivity for an individual Exchange Outlook Web App URL. This example tests the connectivity of a specific Client Access server Contoso12 and tests all Exchange Outlook Web App virtual directories that support Exchange mailboxes. These include the virtual directories that don’t require SSL.

Test-OwaConnectivity -ClientAccessServer:Contoso12 -AllowU
nsecureAccess

Test-PopConnectivity

Use the Test-PopConnectivity cmdlet to verify that the POP3 service is running as expected. The Test-PopConnectivity cmdlet can be used to test the POP3 functionality for a specified Client Access server for all mailboxes on servers running Microsoft Exchange Server 2010 in the same Active Directory site. This example tests the POP3 connectivity of the specific Client Access server Contoso12 and tests all Exchange mailboxes.

Test-PopConnectivity -ClientAccessServer:Contoso12

Test-ImapConnectivity

Use the Test-ImapConnectivity cmdlet to verify that the IMAP4 service is running as expected. The Test-ImapConnectivity cmdlet can be used to test the IMAP4 functionality for a specified Client Access server for all mailboxes on servers running Microsoft Exchange Server 2010 in the same Active Directory site. This example tests the IMAP4 connectivity for the Client Access server Contoso12 by using the credentials for the user contoso\kweku.

Test-ImapConnectivity -ClientAccessServer:Contoso12 -Mailb
oxCredential:(Get-Credential contoso\kweku)

Test-WebServicesConnectivity

Use the Test-WebServicesConnectivity cmdlet to perform basic operations to verify the functionality of Exchange Web Services on a server running Microsoft Exchange Server 2010 that has the Client Access server role installed. This example tests Exchange Web Services on the local Client Access server and allows the test to use an unsecured connection that doesn’t require SSL. A default test account is used.

Test-WebServicesConnectivity -AllowUnsecureAccess

Test-Mailflow

Use the Test-Mailflow cmdlet to diagnose whether mail can be successfully sent from and delivered to the system mailbox on a computer that has the Mailbox server role installed. You can also use this cmdlet to verify that e-mail is sent between Mailbox servers within a defined latency threshold. This example tests message flow from the server Mailbox1 to the e-mail address john@contoso.com.

Test-Mailflow Mailbox1 -TargetEmailAddress john@contoso.com

Many of these tests can be performed using the https://testexchangeconnectivity.com website, but using the EMS lets you keep the tests internal, and doesn’t require you to submit credentials to an external webserver. Depending upon the tests run, you might need to provide the user’s credentials, so it can be useful to have a test user mailbox on each mailbox server.

While this covers the commands you will probably use most often, there are many others available to you. You can see all of the troubleshooting cmdlets available by entering this command in the EMS.

Get-help test-*

You can also read more about related commands at the Microsoft TechNet articles http://technet.microsoft.com/en-us/library/aa998005.aspx or http://technet.microsoft.com/en-us/library/aa998225.aspx.

10 Great Commands for Troubleshooting Client Access Issues

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Every so often I come across a really great tool that I want to add to my toolbox, and share with others. Sometimes, the tool was built with one purpose in mind, but I use it for something else. Microsoft developed the Office 365 Deployment Readiness Tool for organizations to use when they want to move their email, IM, and SharePoint environments up to the cloud with Office 365, and this tool is invaluable for determining whether a company is ready to do this or not. It can be used for more than just that however. Whether you are considering a move to Office 365, another cloud provider, or just want to take a closer look at your internal environment, this tool can give you a very in-depth look into your environment.

The Office 365 Deployment Readiness Tool is a free download from the Office 365 website, using this link. At just over 7 MB, it should download and install fairly quickly on any currently supported workstation or server platform. The only rights needed to run this against your Active Directory are that you use a domain joined computer and a domain account – it does not require admin rights in AD.

The tool launches as soon as you run it, and performs several checks on your forest, and can take from just a few minutes to an hour or more, depending upon how large your environment is and what kind of connectivity you have to a GC. The screen refreshes frequently, and you will see green checkmarks as each test is completed. Once it is done, the Deployment Readiness Tool will report on several aspects of your forest, including:

1. AD infrastructure – including all domains, forests, and trusts
2. The number of objects that will be synchronized to Office 365
3. Forest and domain functional level, and schema extensions
4. Tests for sAMAccountName, UPN, illegal characters, and duplicates
5. Issues with accounts that require remediation, such as spaces in names, illegal characters, or UPN suffixes that are not routable on the Internet
6. An estimate of the number of workstations not ready for Office 365
7. Connectivity tests
8. DNS tests
   and more

click for full size

Results are saved to a html file and include links to detailed results, the deployment guide, and where needed, MSDN articles to assist with understanding and remediation.

While this tool is built to help companies get ready to migrate to Office 365, it could be useful for any migration, whether to another hosted provider or just a new on-premise solution. It can also provide you useful information to create a baseline of your environment. Whether you are taking over for a past administrator, are a consultant for a new client, or an auditor, the information gathered by the Deployment Readiness Tool will be useful, providing you a detailed picture of the environment.

Lync and SharePoint Online prerequisites are checked, as are all mailboxes for any size issues that might impact a migration. Since the idea behind this tool is to assess readiness for Office 365, connectivity tests to the Office 365 service are run. This won’t be as useful to you if you have no plans for moving to the cloud, but will be very important if you do.

If you have even considered in passing moving some of your email services to a cloud services provider, download and run this free tool if only to get an idea of whether or not there are any showstoppers in your environment. If you just inherited a poorly documented email infrastructure as part of a new job, run this tool to get a useful overview of what you are working with. In either case, the information is useful, can be incorporated into documentation and management reports, and contains data you need to know.

The Office 365 Deployment Readiness Tool

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

If you have ever needed to find a ‘missing email’ for a user, then you have probably tried to check queues, consulted log files, and suggested that users look in their outbox, their sent items, and their junk mail folder; all to no avail. Exchange includes a great tool for checking on email messages called Exchange Message Tracking. This browser based tool can be accessed through OWA, and gives the Exchange admin a simple and quick to use interface for checking on email messages, whether they were sent from an internal user or to them from an external sender.

Message Tracking can be accessed either by launching the tool from the Exchange Management Console Toolbox, or from Outlook Web Access. The Toolbox takes you directly to the tool’s interface, while using OWA requires you to choose Options, Manage My Organization, Mail Control, Delivery Reports. Whichever way you choose to get there, you will need Exchange admin rights to check on messages for all users.
The browser interface requires you to select the mailbox that is either the sender or the receiver first. Then, you can run a search based on:

• Search for messages sent to:
• Search for messages received from:
• Search for these words in the subject line:

When searching for results that will include external users as either the sender or the recipient, you will still click the “Select users…” button. You can choose contacts from the GAL, or enter free-form email addresses in that dialog box. It’s a little confusing at first, but easy enough to use once you understand what you need to do. Also, you can search based either on the sender or the recipient; not both.

Once you have entered the criteria, the wizard will check the mailbox and provide the search results in the pane below. Select a message, and then click the details button to learn the status of the message. Messages delivered to internal mailboxes will provide much more information than those sent to external addresses, and can include the date and time the message was delivered, as well as the FQDN of the external system that delivered the message to your Exchange system. Outbound messages are not as data rich; about all it can tell you is the date and time the message was transferred to an external system. Final delivery, hostname, and other details are lacking.

The best part of the message tracking is that you can email the results at the click of a button. Since you are probably checking on the status of a message for some user, sending them the results is a great way to provide them with the information they requested. So the next time a user wants to know what happened to a message, use Exchange’s Message Tracking to let them know.

How to find that missing email with Exchange Message Tracking

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

How familiar is this? A user opens a help desk ticket (or calls you directly) because they sent a proposal to a customer ‘ten minutes ago’ and the customer still hasn’t received it; or a user ends up using a Hotmail account because Outlook is so slow. Ringing any bells? Email typically works within seconds and users are accustomed to that speed. When anything takes longer than clicking Send/Receive All, they tend to assume something is broken and act accordingly. While we know that there are countless things that can add a few seconds to how long it takes a message to get from A to B, sometimes there really is a problem somewhere along the way, and we need a tool to figure out where that problem is. Enter the Exchange Mail Flow Troubleshooter.

This tool is a separate binary from the EMC, but by integrating its launcher into the EMC, administrators can easily find and use the tool. While it is primarily an internal testing tool, it can troubleshoot some problems with remote systems, especially when the delays involve your users sending mail to external recipients.

To launch the tool, use the Exchange Management Console to browse down to the bottom of the console tree and select Tools. In the Toolbox Window select Mail Flow Troubleshooter, then in the Actions Pane click Open Tool. The Mail Flow Troubleshooter will launch, and the first thing it will do is check for updates from the Microsoft website. Any updates will be applied, and then you are ready to go.

As with most Exchange GUI tools, this one is wizard driven. The welcome screen first prompts you to create a title for the session. I like to use the name of the user experiencing trouble, followed by the date. Then you have to choose the mail flow issue you want to troubleshoot. This tool can help with:

• Users are receiving unexpected non-delivery reports when sending messages,
• Expected messages from senders are delayed or are not received by some recipients,
• Messages destined to recipients are delayed or not received by some recipients,
• Messages are backing up in one or more queues on a server,
• Messages sent by users are pending submission on their mailbox server(s).

The first five are version agnostic, so they can troubleshoot any legacy Exchange system still in the environment, while the last one only works with Exchange 2007 and 2010 mailbox servers. When you pick the option that best describes the mail flow issue you are dealing with, you can then select to hide details if you want to, then you click next. Depending on the issue, you will need to provide the name of the Exchange server generating the error, hosting the user’s mailbox, and/or that is the last Exchange server in the path before email hits the Internet, and information about the sender and receiver SMTP address. The next few pages run through information gathering and diagnostics (e.g. DNS lookups, AD, WMI queries, connectivity checks as appropriate), and then it will attempt to submit a message to analyze the results of that test.

Even if one of the earlier steps in a test reports an issue, you want to run through all steps to completion to be sure you have checked all the possible issues that might contribute to the problem. When done, the tool will provide detailed results from each test run, and where appropriate, suggestions for corrective actions. Reports can be viewed in list or tree view, printed, or saved as XML, HTM, or CSV format so that you can share them, or keep them for future reference.

The next time a user reports a mail flow issue, use this tool to determine the cause and appropriate solution. You can save yourself hours of troubleshooting wild goose chases and generate some great reports for any request for outage you have to file.

How to Solve Exchange Issues with the Mail Flow Troubleshooter

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

If you are reading this blog post, you either are, or want to be, an email administrator. It stands to reason, right? I’ve been working with email systems for more than a decade, and have found great email admins, average email admins, and folks I wouldn’t let admin an AOL account. Knowing your email system of choice, be it Exchange, or Notes, or SendMail, or any of the others, is obviously important; but that is just the start of being a truly great administrator. There are several other areas you should be competent in. The following list is what I feel good email admins should know, based on my own experiences. Agree or disagree, if you are an experienced email administrator I would love for you to comment on what you think are good skills to have or those you think I missed the boat on.

1.     DNS

There are so many ways an email infrastructure depends upon DNS, I cannot believe any email admin doesn’t have a 400 level understanding of DNS, but I see it every day. I don’t expect you to be able to create SPF records off the top of your head, but I do expect you to know what MX records actually do, and that weights make a difference.

2.     Networking

You should be able to take and analyze a network trace, using whatever tool you want. You should know the difference between TCP and UDP, what ports your clients need and your servers need for each of the major mail protocols, and how to filter on them. You should also have an idea of how firewalls work and what ports a firewall needs to allow in each direction to support your traffic.

3.     Certificates

There are a lot of uses for certificates within email. Whether you are using certificates to secure SMTP, POP3, and IMAP using SSL/TLS, or certificates for signing and encrypting email using S/MIME, you should have a basic idea of how certificates work, how to generate CSRs, enroll the certs into your applications once they are issued, and how to troubleshoot them.

4.     Storage

Even the best run email system will run afoul of storage problems eventually if for no other reason than that users a packrats who will never delete anything. Understanding storage doesn’t mean I think you should be a SAN expert, but you do need to know how to baseline your storage needs, monitor consumption trends so you can address problems before they occur, and the basics of disk i/o and what your servers need to meet user demand.

5.     Server operating system

Your email server depends upon the underlying operating system, so you need to know at least as much about how your Windows or Linux o/s works; how to troubleshoot it, how to optimize it, and how to patch it.

6.     Active Directory

Exchange is absolutely dependent upon Active Directory, and many other email packages will use AD for authentication. You need to understand the dependencies, especially Exchange’s interactions with Global Catalog servers; how replication works and how it behaves in your domain(s), and the difference between distribution lists and mail enabled security groups. Group Policy-not so much.

7.     The clients

I admit, I still struggle with parts of this one, but since users see the email client every day, and will (hopefully) never see the server side of your email infrastructure, you need to know how to support the current version client. The Help Desk can be responsible for things like mail merges and stationary, but you need to know how to connect the client to the server using every possible connection method, and be able to troubleshoot things like offline address books, folders, restoring deleted objects, etc.

8.     SMTP, POP3 and IMAP

Many Exchange admins I have worked with have no problem with the MAPI connections’ Outlook uses, and are comfortable with Outlook Web Access, but when they have to support a legacy client that can only do POP3 or IMAP with SMTP, they get lost in the weeds. Whether it is an application, a scanner, or a monitoring system, being able to provide services to these ‘legacy’ clients will remain important for many years to come.

9.     People skills

Remember Stephen Root’s character, Milton Waddams, from Office Space? If you don’t, do nothing else until you go see that movie. You owe it to yourself to see that movie, then you will understand what I am about to say. I have been to far too many clients where the email admin was described to me as a Milton. Their physical appearance had nothing to do with this; it’s how they interacted with others. Email is arguably the most important system within all of IT, and as a rock-star email admin, you do your job best when email works flawlessly. That unfortunately means you won’t be in the limelight too often. When email goes down, it’s a code brown situation and the end of the world all at once, which means grace under pressure, and unflappable mien, and the ability to talk to anyone from the part-time clerk to the CEO may be required. If you ignore all the others on this list, trust me on this one. Learn how to to work well with others.

If you possess skills in each of these eight areas, you should be well positioned to troubleshoot any and all problems that might come up with your system. You don’t need to be expert in all of them; there will be plenty of times you need to call on a specialist, but having the understanding and the vocabulary will enable you to help them quickly resolve even the most complex of issues.

Once again, those of you who have been doing this for a while and have been around the block more than twice; what other skills do budding admins need, and do you think I am off base with any of the ones I listed?

Nine Skills All Email Admins Must Have

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

One of the toughest jobs for an email admin is troubleshooting ambiguous issues. These can be intermittent problems with connectivity, non-specific errors in the logs, or the dreaded user report that things are “just slow”. Performance based issues are particularly troubling because they are so often subjective in nature. The user didn’t observe an error pop-up on their screen; they cannot tell you that the system is 150% slower to transfer the same 5K email message as it did yesterday at this time; the queue might be backing up, but frequently there is no code that states why.

Fortunately, Exchange 2010 has a great tool built into the Management Console for troubleshooting performance issues called the Performance Troubleshooter. You have to hand it to those folks; their names are original. The Performance Troubleshooter can assist an Exchange admin with determining the root cause, as well as the resolution for, several different performance related problems tied to RPC connection issues with the Exchange system, including:

• Multiple users are complaining of delays while using Outlook, or are seeing the cancellable RPC dialog frequently
• The number of RPC operations per second is higher than expected
• The number of outstanding RPC requests is high

You can access this tool in the Exchange Management Console by browsing down to the Toolbox, and then launching the Performance Troubleshooter. The tool launches and the first thing it does is check for updates, which is a great way to be sure you are using the most up to date tools. It then asks you if you want to start a new troubleshooting session or resume a previous one, and assuming you choose a new session, takes you to the screen where you select one of the above choices.

The tool will ask you to specify a GC, and the Exchange server you want to test, and then run through a series of test and log gathering. Make sure you are running it as a member of the Exchange Admins group. The wizard will prompt you for a place to save its log file, and then run a series of diagnostics designed to find any performance bottlenecks or configuration issues. It will check the GC logs, the Exchange server performance logs (including the Function Call Log), and event logs, and run some health checks on the servers looking for common causes of the issue. It then presents the results in a tabular output for Overview, Performance Issues, Execution Issues, and General Information, and lets you export the report in XML. CSV, or HTML format. Problems are called out visually, and recommendations are provided for resolving the issues.

As an aside, if your users are accessing PST files over the network, or you do not have enough Global Catalog servers in proximity to your users, you will see the cancellable RPC dialog all over the place. The wording is a little different depending upon whether the client is doing a GAL lookup or just hammering a file server trying to access a network PST, but the result is the same.

By running this tool once a month, you can spot any developing issues before they become major concerns, and establish a baseline for your servers. If you are experiencing problems with RPC connectivity, this can find the cause in a matter of minutes and provide you with good advice on how to resolve it, saving you hours of log parsing and KB article reading. Check the tool out, and add it to your repertoire of Exchange tools.

How to Troubleshoot Exchange RPC Performance Issues

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

I am normally one to go rush out and deploy patches and updates at my earliest convenience; I have been burned far more often by unpatched machines than by bad patches, but it seems that in this case time was on my side, as I had not yet deployed RU4 to my Exchange environment. Truth be told, it has been a very busy couple of weeks and I wasn’t even aware that RU4 was out, which strictly speaking, it isn’t, as Microsoft has pulled this RU from the shelves after some customers reported issues with it.

It seems that RU4 causes issues with some customers when a client uses Outlook to copy or move a folder. Contents of the moved folder are deleted. Fortunately, the deleted content can be recovered from the Recoverable Items folder, so no permanent data loss has hit anyone, but I’m sure many end users are unaware of the Recoverable Items folder, and would not have checked there even if they were.

If you have not yet deployed RU4 in your environment, don’t! Continue to test and deploy monthly patches that are relevant to your systems during the monthly patching cycles, and wait until RU5 comes out for your next RU. That is expected sometime in August.

If you have already deployed RU4, there is no need to panic. You can obtain an Interim Update from Microsoft to fix this issue. Contact Microsoft Support using the appropriate contact number for your country, and reference KB 2581545. Note: that KB link is not valid yet, as the KB article has not been released yet, but it will be published soon. I have also seen scattered reports that some customers are having problems getting this patch when they phone into their support contact. If your support contact is not able to locate this interim update, politely ask them to escalate your call. This is a relatively young issue, and first level support contacts may not be aware of it yet. Escalating your call will get to someone who is aware, and they will get you the update.

The interim update will need to be applied to all Client Access Servers and Mailbox servers to which you applied RU4. If you have applied RU4 to Hub or Edge Transport servers, you don’t need to worry about applying this interim update to them. Again, if you have not yet applied RU4 to your environment, don’t. Wait for RU5 and continue with your normal patching procedures for monthly updates.

Exchange 2010 SP1 RU4 pulled after customer issues

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

A few months ago I wrote a post listing fourteen online resources for email admins that included several of my favourite troubleshooting resources. In this post, I want to take you for a closer look at the best one of the lot for testing remote connectivity to Exchange, the Microsoft Exchange Server Remote Connectivity Analyzer. You can access this test suite by clicking the link above, or directly at its URL, https://www.testexchangeconnectivity.com/. There are several great tests this tool can run through to ensure that you have properly set up remote access to your Exchange infrastructure, and you will want to bookmark this site and refer to it whenever you setup, or change, the external connectivity to Exchange.

Before you begin, create an unprivileged test account in your Active Directory, and make sure it has a valid Exchange mailbox. You can of course use your own account or anyone else’s, but this site requires that you enter valid user credentials, and it’s a best practice not to submit valid credentials for a ‘real’ user to an external site outside of your complete control. If you want to skip that step, that’s on you, but I always keep a test account handy for things like this.

Once you have your test account ready, take a look at the site to see what it offers. There are four categories with two tests each:

1. Microsoft Exchange ActiveSync Connectivity Tests
   Exchange ActiveSync
   Exchange ActiveSync Autodiscover
2. Microsoft Exchange Web Services Connectivity Tests
   Synchronization, Notification, Availability, and Automatic Replies (OOF)
   Service Account Access (Developers)
3. Microsoft Office Outlook Connectivity Tests
   Outlook Anywhere (RPC over HTTP)
   Outlook Autodiscover
4. Internet E-Mail Tests
   Inbound SMTP E-Mail
   Outbound SMTP E-Mail

The ActiveSync Connectivity tests can validate your DNS records, as well as how you have exposed EAS connections to the Internet (through Microsoft TMG or other reverse proxy, or by passing HTTPS traffic through to your CAS server directly). Both of these tests will in essence configure a mail client using EAS, and requires that valid test account to connect all the way through. In case you are using self-signed certificates, it even gives you the option to not validate certificates.

The EWS tests are useful for admins who need to support Entourage or other applications that require access through Exchange Web Services, and can verify the ability to create/delete messages and other service activities.

The Outlook Connectivity tests basically configure an Outlook client using the RPC over HTTP protocol. It can also validate all your DNS records, whether you are using A or SRV for autodiscovery. See this post for more on Autodiscover.

The Internet E-Mail tests can send a test message to your account from an external sender, and can also confirm your DNS records for MX, PTR, and Sender ID, and make sure your host is not listed on any DNS Reverse Blacklist service.

While all of these could be done using your external Hotmail account, and one or more systems connected to a DSL circuit external to your corporate network, it’s really useful and a great timesaver to have all eight tests available to you with nothing more required than a web browser and a test account. Even if you have a working system now, take these eight tests for a spin to see how things you might not be able to test, like Mac clients, would function, and also to see how your DNS records test. You might be surprised at what you find out. If you pass all eight the first time through, you’ve earned bragging rights; leave a comment and let me know.

How to troubleshoot remote connectivity to Exchange

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

There are several components within Exchange 2010 that administrators may not come across unless they are troubleshooting very specific issues, or reading level 400 texts. This is the first post in what may become a series on some of the deeper internal workings of Exchange 2010. In this first post, I want to introduce you to the Transport Dumpster.

The Transport Dumpster is just one of the inner components of Exchange that sits quietly and unobtrusively on a Hub Transport server, helping to protect data in the event a server experiences any type of failure that would otherwise lead to a loss. You may not notice it, but you will be glad it is there, and in certain circumstances, you may want to adjust its default settings.

The Transport Dumpster first appeared with Exchange 2007, and is a queue that can be found on every Hub Transport server that resides in an Active Directory Site that contains a DAG. It helps prevent loss of data sent from the Hub Transport Server to a Database Availability Group when something fails that would result in a lost message. There is one Transport Dumpster per replicated database.

When a message is sent from a Hub Transport server to a Mailbox server, the messages is stored in the Transport Dumpster queue for the DAG until the Hub Transport server receives a notification that the transaction logs for a particular message have been replicated to all copies of the DAG and inspected by the Mailbox server. Once every DAG member confirms that the message logs have been committed, the message will be purged from the queue, but if a DAG member either does not acknowledge the logs, or reports that the logs failed, the Transport Dumpster can resend the information to the DAG member.

The Transport Dumpster stores messages for a period of time controlled by the MaxDumpsterTime setting, which is seven days by default, or when the maximum size of the queue is reached. The default for that is 18MB, but Microsoft recommends that this is set to 1.5 times the maximum message size limit. Whenever one or the other limit is reached, messages are purged on a first-in, first-out basis. You can view the current settings using the EMS and running the command

Get-TransportConfig | fl *Dumpster*

You can adjust the parameters for the Transport Dumpster by using these commands in the EMS. In these examples, we set the size to 30 megabytes, and the lifetime to 4 days.

Set-TransportConfig -MaxDumpsterSizePerDatabase 30MB

Set-TransportConfig -MaxDumpsterTime 4

You can enter the size specifying MB for megabytes, or you can enter a value in bytes. The time can be entered in EnhanceTimeSpan format, but a simple digit will give you days.

It is important to note that the Transport Dumpster does not protect against data loss during a failure should the message be destined for either a public folder, or for a mailbox database that is not a part of a Database Availability Group. It can only protect replicated mailbox databases.

If you found this post interesting/useful, please leave a comment to let me know, or tweet it if you’d prefer. If there is enough interest, I will continue the series on the inner workings of Exchange.

Inside Exchange 2010: What is the Transport Dumpster?

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Communications between computer systems can occur synchronously or asynchronously. Some of the benefits of asynchronous communications are that they are faster, can run over lower bandwidth technologies and require less maintenance of the connection.

But when a more reliable connection is needed then synchronous communications are used. Such synchronous communications include Remote Procedure Calls (RPC) and the Distributed Component Object Model (DCOM). But because there is more hand shaking going on with synchronous communications it is generally not as fast as asynchronous communications. An alternative solution for email communications is to use an asynchronous programming model such as Message Queuing.

Microsoft Message Queuing (MSMQ) operations provide several messaging features including: authentication, encryption, dead-letter queues, security, and other basic features. If any of these features have problems then Message Queuing operations in general will also have problems.

Microsoft Message Queuing works very well in a distributed environment made up of a variety of applications. Applications running on different systems can communicate across the network and communicate with systems that are not online at the moment. The technology uses queues for the storing of data. The message queues are used by a message service to forward the data onto another queue. Then a “receiver” application pulls the data from the second queue. At this point the data is then processed. The technology is facilitated by multiple sending and receiving applications.

One of the benefits of using message queues is that communication is not broken even though systems may be down for short periods of time. Messages in the queues are not lost even though a system, or systems, is unable to receive communications. Client applications send their message data and then are free to continue their own processing independent of any problems between systems. Once a server queue becomes available then the message queue service on the client can forward the message data to the server queue for processing by the receiver application. This technology is known as a store and forward technology.

However, it is not without occasional errors. Sometimes the communication operations will result in MSEXchangeRepl error messages on the Exchange Server such as the following:

Source: MSExchangeRepl
Event ID: 2163
Task Category: Service
Level: Error
Description:
The log copier for database ‘DB’ received an error from the source server ‘server’: File ‘Drive:\Logs\#######.log’ could not be opened. The process cannot access the file because it is being used by another process. The copier will automatically retry after a short delay.

This error can occur when a log from a source has not yet been closed or flushed and the log copier component is trying to copy it. The log has to be closed for the copy operation to succeed. The event is generated because new log files are being polled instead of waiting for the source logs to complete their close-flush-notify operation.

Administrators should note that Event ID 2163, whose source is MSExchangeRepl, is not a cause for concern as per Microsoft web site and can be ignored.

As mentioned earlier if there are problems with any of the features of Message Queuing then there will be problems in general. More troublesome issues can result in failure such as is shown in this event description for Event ID 2163:

Product:     Windows Operating System
ID:     2163
Source:     MSMQ
Version:     6.0
Symbolic Name:     QM_SERVICE_STOPPED
Message:     The Message Queuing service stopped.

Note that the source for this event is the MSMQ service itself. There are several reasons for the Message Queuing service to stop. The best place to look for what may be the cause of this failure is to check the event logs. There may be an issue with the start up of the service which can result in the failure message. Once those issues have been identified and fixed then the MSMQ service will need to be restarted.

Administrator privilege is required in order to restart the service. The MSMQ service can be restarted with the following steps:

1. Open the Services snap-in.
2. Click Start.
3. Type services.msc in the search box.
4. Press Enter.
5. Right-click Message Queuing.
6. Click Restart.

Note that all dependent services must also be restarted.

Lastly, an administrator should verify that the MSMQ Service is installed and running. As with the procedure to restart Message Queuing, the operator will need Administrator privileges or an appropriate authority level.

A successful restart of the MSMQ Service can be performed using the following steps:

1. Open the Services snap-in.
2. Click Start.
3. Type services.msc in the search box.
4. Press Enter.
5. Locate the Message Queuing service.
6. The value in the Status column should be displayed as Started.

Troubleshooting Message Queuing

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Occasionally, administrators will receive error messages in the application log that are more generic in their descriptions of the problems than an administrator would like them to be. It is so much easier to troubleshoot a problem when there are more details available associated with the received error.

One such error message which does contain details is the error message associated with Event ID 12014. The error message will contain many details and resembles the following text description:

“Microsoft Exchange could not find a certificate that contains the domain name mail.server.com in the personal store on the local computer. Therefore, it is unable to support the STARTTLS SMTP verb for the connector Internet with a FQDN parameter of mail.server.com. If the connector’s FQDN is not specified, the computer’s FQDN is used. Verify the connector configuration and the installed certificates to make sure that there is a certificate with a domain name for that FQDN. If this certificate exists, run Enable-ExchangeCertificate -Services SMTP to make sure that the Microsoft Exchange Transport service has access to the certificate key.”

Event 12014 is a Warning event that indicates that a problem occurred while loading a certificate to be used for STARTTLS. This problem generally occurs if one or both of the following conditions is true:

1. The fully qualified domain name (FQDN) that is specified in the Warning event has been defined on a Receive connector or Send connector on a Microsoft Exchange Server 2007 transport server. In addition, the same computer that contains the FQDN in the Subject, or Subject Alternative Name fields, does not have a certificate installed.
2. There may be a third-party or custom certificate installed on the server which contains a matching FQDN. And the certificate has not been enabled for the Simple Mail Transfer Protocol (SMTP) service.

One of the Transport Layer Security (TLS) requirements is that a valid certificate must be installed in the system’s personal certificate store.

In order to begin troubleshooting the server, the account that an administrator uses must be delegated the following:

• Exchange View-Only Administrator role to run the Get-ExchangeCertificate cmdlet
• Exchange Server Administrator role and local Administrators group for the target server to run the New-ExchangeCertificate cmdlet or the Enable-ExchangeCertificate cmdlet

Additionally, if the server has the Edge Transport server role installed then to run any of these cmdlets on the system, an administrator must log on by using an account that is a member of the local Administrators group on that system.

An administrator should first review the configuration of the certificates that are installed on the Exchange server and the configuration of all Receive connectors and Send connectors that are installed on the server. An administrator can use the following commands to view the configuration:

1. Get-ExchangeCertificate | FL *
2. Get-ReceiveConnector | FL name, fqdn, objectClass Get-SendConnector | FL name, fqdn, objectClass

In order to display the services that are enabled for the installed certificate, an administrator must use the asterisk (*) when they run the FL argument on the Get-ExchangeCertificate cmdlet. The services values will not display if the * is not specified in the task parameters.

After running the above commands, the FQDN that is returned with the Warning event should be compared with the FQDN that is defined on each connector and with the CertificateDomains values that are defined on each certificate. The CertificateDomains value is a concatenation of the Subject and Subject Alternative Name fields on the certificate.

An administrator should verify that each connector that is using TLS has a corresponding certificate that includes the FQDN of the connector in the CertificateDomains values of the certificate. Make a note of any connectors that are enabled for TLS but do not have a corresponding certificate where the FQDN of the connector is in the CertificateDomains values of the certificate.

The Services value on each certificate should also be inspected. Note that a certificate for TLS must be enabled for the SMTP service that uses a Services value of SMTP.

If the FQDN is not listed on the CertificateDomains parameter, then an administrator must create a new certificate and specify the FQDN of the connector that is returned in this warning message. The certificate can be created by using the New-ExchangeCertificate cmdlet. A third-party or custom certificate may also be used. You can use the New-ExchangeCertificate cmdlet to generate the certificate request.

Lastly, if a third-party or custom certificate has been installed on the server and the certificate contains a matching FQDN but is not yet enabled for the SMTP service, then it must be enabled for the SMTP service.

Troubleshooting Event ID 12014

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators