While there are other options for connecting Exchange to the Internet, most companies will probably go with placing an Exchange server in the DMZ and deploying the Edge Transport role. When you do this, you typically will subscribe this server to your internal Exchange infrastructure using an edge subscription. While this is an excellent way to maintain a bastion host in the DMZ while still managing it through your internal Exchange Management Console (or Exchange Management Shell,) edge subscriptions can be very tricky to troubleshoot when things go bump in the night.

Microsoft has published a very useful, if somewhat complex, flow chart that administrators can follow if they find themselves needing to troubleshoot edge subscription problems. Found here, it presents a methodical, and step by step process, for starting at the edge transport server, and working your way through the required connections to get to the hub transport server. The flow chart also takes into account that you will have a Microsoft TMG server infrastructure in place between the edge and hub transport roles.

There are also supporting instructions on the five major areas that require more than casual investigation. These are

1. How to check if Edge Subscription is configured
2. How to enable connectivity for EdgeSync traffic
3. How to check system policy rules
4. How to create and import an Edge Subscription file
5. How to verify that all Forefront TMG Managed Control Service services are running

Verifying the services is a very important area to look at, as most TMG admins will assume that the services tab in TMG would show you all the running services. For Microsoft TMG, this is not actually the case. The following table shows the services that are managed in TMG or Computer Management, though all can be managed at the command line as well.

There are two areas in this troubleshooting that I cannot stress enough. First, ensure that your DNS and NetBIOS name resolution are working correctly in both directions. The Edge Transport server, not belonging to your domain, and being in a DMZ, may not be able to directly register its A and PTR records in your DNS. If you are scavenging, those records could be deleted and that will bring things to a screeching halt very quickly. And there is plenty of NetBIOS still going on between Windows systems even today. It’s worth the extra effort to implement a WINS infrastructure, and add static records for any DMZ hosts that cannot register themselves.

Second, make sure your certificates are correctly set up, and trusted by all systems. If you use an Active Directory integrated CA, then your internal servers will trust any certificates issued by the PKI automatically, but you will have to manually import the root CA certificate onto the Edge Transport server. You may also want to make sure that all the certificates in use cover all the service names required. See this post on how to easily create CSRs for all your Exchange needs.

What tricks do you have for troubleshooting Exchange?

How to Troubleshoot Edge Subscriptions

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

One of the most important services that Exchange Server runs is the MSExchangeTransport service. This service provides the SMTP transport functions. If this service fails or is halted or does not start then mail will not be transmitted. All Exchange Hub Transport and Edge Transport server roles have this service.

There are a variety of reasons for the MSExchangeTransport service to fail. As with all troubleshooting processes it is best to start with the event logs. Error messages such as the following can be found in the system log:

Date: 2008/04/05
Event ID: 16023
Source: MSExchangeTransport
User: N/A
Type: Error
Category: Configuration

Microsoft Exchange couldn’t start transport agents. The Microsoft Exchange Transport service will be stopped. Exception details: Invalid agent configuration in file ‘E:\Program Files\TransportRoles\Shared\agents.config’. : Microsoft.Exchange.Data.ExchangeConfigurationExcep tion: Invalid agent configuration in file ‘E:\Program Files\TransportRoles\Shared\agents.config’. —> System.Xml.XmlException: Root element is missing.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseDocumentContent( )
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.Load(Stream inStream)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExConfiguration.Load(String filePath)
— End of inner exception stack trace —
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExConfiguration.Load(String filePath)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExRuntime.Initialize(String configFile, String agentGroup)
at Microsoft.Exchange.Transport.Categorizer.MExEvents.Initialize(String configFilePath)
at Microsoft.Exchange.Transport.Components.CategorizerMExRuntimeLoader.Load()

———————————————————————————————————————————————————————————

Date: 2008/04/05
Event ID: 7001
Source: MSExchangeTransport
User: N/A
Type: Information
Category: Components

The service will be stopped. Reason: Invalid agent configuration in file ‘E:\Program Files\TransportRoles\Shared\agents.config’.

————————————————————————————————————————————————-

Date: 2008/04/05
Event ID: 1022
Source: MSExchangeTransport
User: N/A
Type: Information
Category: ProcessManager

Worker process with process ID 4268 requested the service to be stopped.

————————————————————————————————————————————————-

Date: 2008/04/05
Event ID: 1002
Source: MSExchangeTransport
User: N/A
Type: Information
Category: ProcessManager

The service is trying to stop.

————————————————————————————————————————————————-

Date: 2008/04/05
Event ID: 1003
Source: MSExchangeTransport
User: N/A
Type: Information
Category: ProcessManager

The service has stopped.

————————————————————————————————————————————————-

A solution to this problem is to restore the defaults by running the “ReinstallDefaultTransportAgents.ps1” file from the exchange management shell. The problem occurs because a valid “agents.config” file can not be found in the location “C:\Program Files\Microsoft\Exchange Server\TransportRoles\Shared\agents.config”. The default configuration for transport agents can be recovered by running “scripts\ReinstallDefaultTransportAgents.ps1″.

Another possible solution is that the permission on the TransportRoles folder may be set incorrectly for the Network Service. The Network Service needs full permission on the TransportRoles folder and children.

In addition to the error messages listed above another condition can occur in which the Hub Transport Role Fails. The error message that will be generated is as follows:

Hub Transport Role
Failed
Error:
Service ‘MSExchangeTransport’ failed to reach  status ‘Running’ on this server.

What this means is that the Microsoft Exchange Transport service continuously tries to start but fails each time. If this happens then a workaround is to install the IPv6 protocol. This may have been turned off if a static IP address was chosen when Exchange server was installed. The problem is that the send connector will not work. After enabling IPv6 on the network interface card in use on the server an administrator may still need to assign an address to IPv6.

Another MSExchangeTransport service failure event that can be generated is referenced in the system log by the following error message:

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7001
Date: 11/25/2008
Time: 8:16:12 AM
User: N/A
Computer: Servername
Description:
The Microsoft Exchange Transport service depends on the FSEIMC service which failed to start because of the following error: The dependency service or group failed to start.

This error condition can be fixed by re-installing Forefront Security for Exchange Server Service Pack 2 (SP2).

Another condition which prevents the MSExchangeTransport service from starting on the Exchange Server Hub is when there is a configuration problem. A configuration error is indicated in the application log by the following message:

Event Type: Error
Event Source: MSExchangeTransport
Event Category: Routing
Event ID: 5023
Description:
A transient configuration error was detected while the routing configuration was loading. Exception details: The local server isn’t a member of any routing group. : Microsoft.Exchange.Transport.Categorizer.TransientRoutingException: The local server isn’t a member of any routing group. At Microsoft.Exchange.Transport.Categorizer.RoutingGroupRelayMap.ValidateConfig(ExchangeTopology topology, RawRoutingConfigData rawConfigData, DateTime timestamp)at Microsoft.Exchange.Transport.Categorizer.RoutingTables.ValidateConfig(RawRoutingConfigData rawConfigData)at Microsoft.Exchange.Transport.Categorizer.RoutingTables.PopulateTables(RawRoutingConfigData rawConfigData)

The error occurs because there is no value set for the msExchHomeRoutinggroup attribute on the Exchange 2007 Hub server object for the Active Directory directory service. This error can be fixed using the ADSI edit tool and entering the “distinguishedName” value for the msExchHomeRoutingGroup attribute. A restart of the Microsoft Exchange Transport service on the Exchange 2007 Hub server will be needed for the change to take effect.

Troubleshooting Failed MSExchangeTransport Service

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Like all servers, email servers are subject to heavy loads and when that happens the servers can slow down to a crawl. And if administrators are not monitoring the performance then they’ll be sure to find out about the slower performance issues from their users.

Email servers can slow down for many reasons such as increased traffic flow or attacks on the server. Sometime it is a result of increased spam messages. Administrators can use the Queue Viewer graphical user interface to view information about messages and various queues such as the delivery queue. Additionally the command line interface for Exchange server can also be used to pull out information. Actions such as modifying the status of queues and messages can be performed individually or in bulk. One or more objects can be selected to perform these actions on.

In Exchange server, queues are used to hold messages before further processing can occur. There are five queues that exist only on Hub Transport servers or Edge Transport servers. Each of these queues is used in the processing of email messages which flow through the server. The processing queues include the following:

• Mailbox delivery queue
• Poison message queue
• Remote delivery queue
• Submission queue
• Unreachable queue

The transport server processes the messages contained in the queues. And access to the message queues is performed by the Exchange Server using the Extensible Storage Engine (ESE) low-level APIs.

Sometimes the queues fill up with many non-delivery reports from the postmaster account. This condition can occur if your computer is the target of a reverse non-delivery report (NDR) attack. This issue can be resolved by creating a recipient filter to prevent Exchange Server 2003 from accepting messages that are sent to recipients who do not exist.

Another problem that can happen is when message flow is slow to the local delivery queue. When this occurs then email messages in the Simple Mail Transfer Protocol (SMTP) queue will get backed up. These email messages that are bound for local delivery end up getting trapped in the SMTP queue. A further undesirable result is that of high disk queue lengths on the Exchange Server’s install drive. This install drive is typically used for the C:\Program Files\Exchsrvr\ directory. These conditions can occur when the information store working folder’s disk drive gets too busy and slows down.

This type of problem can be corrected by pointing the working folder of the information store used for email message conversion to a faster drive. An administrator can follow the steps below:

1. Start – Run – type in “regedit” to start the Registry Editor.
2. Locate the following key in the registry and then select it: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem
3. Double-click Working Directory:REG_SZ:Folder_Path in the right hand pane.
4. Type the path of a folder on a faster drive in the String Editor box.
5. Click OK.
6. Close the Registry Editor.

As always, any time you touch the registry you should first make a backup of the registry. This way, if there is a problem for whatever reason; you’ll be able to restore the registry to a known working point.

A similar problem can occur because of a disconnected SMTP mailbox. This can cause messages to be held in the local delivery queue. Administrators might try to dismount and then remount the information store to correct the problem. This action, however, can result in more event error messages being logged in the application log such as the following:

Event Type: Error
Event Source: MSExchangeTransport
Event Category: Exchange Store Driver
Event ID: 326
Date: date
Time: time
User: N/A
Computer: ServerName
Description:
Service Account failed to logon to the store as
/o=organization/ou=First Administrative Group/
cn=Configuration/cn=Connections/cn=SMTP
(ServerName)/cn={CCCF3BE6-219C-4379-985D-851B766EDDC7}.
Error code : 0x80040148. For more information,
click http://search.support.microsoft.com/search/?adv=1.

Event Type: Error
Event Source: MSExchangeTransport
Event Category: Exchange Store Driver
Event ID: 326
Date: date
Time: time
User: N/A
Computer: ServerName
Description:
Service Account failed to logon to the store as
/o=organization/ou=First Administrative Group/
cn=Configuration/cn=Connections/cn=SMTP (ServerName)/
cn={CCCF3BE6-219C-4379-985D-851B766EDDC7}.
Error code : 0x80040158. For more information,
click http://search.support.microsoft.com/search/?adv=1.

Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Event ID: 1194
Date: date
Time: time
User: N/A
Computer: ServerName
Description:
Accept clients on external interface SMTP
failed with error 0x80004005. For more information,
click http://search.support.microsoft.com/search/?adv=1.

This problem can be resolved – assuming the Active Directory object for the SMTP mailbox is missing – by creating a new mailGateway object in Active Directory. This newly created object will have a new GUID. The next time the store is mounted the information store will create a new SMTP mailbox object.

Troubleshooting Message Queues

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Most Exchange Server environments will grow beyond their original sizing.  Sometimes this growth has been accounted for in the design, and sometimes it is not.  In either case the question to ask as the user base increases is whether to scale up or scale out.

What do each of those terms mean?  An example of scaling up would be increasing the capacity of a server to handle higher loads.  An example of scaling out would be adding new servers to spread the higher load across more hosts.

Each approach has its pros and cons in the context of the different Exchange Server roles.  In this post I’ll discuss the Exchange Server 2007 Transport server roles and the different scaling considerations that they have.

Edge Transport/Hub Transport Servers

The role of the Edge Transport server is to route incoming and outgoing internet email while applying security and compliance rules to messages.  The role of the Hub Transport is basically the same though it is also responsible for routing emails between internal mailboxes as well.

The Transport Server workload relies primarily on fast CPUs to assess and process each email message against these rules, and fast disks for reading and writing messages to the transport queue.

Processor Scaling

Scaling up with more CPU cores to handle higher workloads is appropriate; however, there is a practical limit of 4 CPU cores per Edge Transport server and 8 cores per Hub Transport server.  This is an obvious “scale out” threshold for this server role – if the server is already at the maximum number of cores and CPU is showing signs of a performance bottleneck then it is time to look at scaling out to more Transport servers.

Disk Scaling

Disk performance is slightly different.  Each Transport server has a transport queue, which is a location on the disk in which emails are read and written in a database.  Emails are only temporarily stored in this queue, and large capacity disks are quite cheap and common in servers these days, so disk space is not likely to be a problem for the volume hosting the transport queue.

On the other hand, disk speed may become a problem in larger Exchange Server environments where email traffic is very high.  Because the transport queue can’t be split across multiple volumes within a single server if disk performance becomes a bottleneck the only option is to scale out to more Transport servers.

Scaling Impacts

Scaling up a Transport server usually won’t affect the configuration requirements for the server.  However scaling out often will require some configuration or architecture changes.  Hub Transport servers have their own load balancing algorithms built-in so not much needs to be done there, but Edge Transport servers will need some attention.

Each Edge Transport server needs to be “subscribed” to the Active Directory Site that hosts the Hub Transport servers it will relay messages with.  This subscription is configured per-server.  Furthermore, there is no common settings location for multiple Edge Transport server, each must be configured individually (although this can be scripted to reduce administrative effort).

Finally, once you scale out to more than one Edge Transport server either multiple public IP addresses and MX records are required, or a load balancing solution needs to be implemented for inbound internet email traffic.

In the next post in this series I will discuss scaling considerations for the Client Access server role.

Scaling Up vs Scaling Out Exchange Server 2007 Roles Part 1 – Transport Servers

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Exchange Server 2007 introduced new terminology for describing the domain names that it will accept email for, and what it will do with that email.  This is referred to in Exchange Server 2007 as Accepted Domains.

In Microsoft’s own words, “an accepted domain is any Simple Mail Transfer Protocol (SMTP) namespace for which a Microsoft Exchange organization sends or receives e-mail.”

Accepted Domains fall into one of 3 categories – Authoritative, Internal Relay, and External Relay.  Any given namespace that is an Accepted Domain can be only one of those three types.

Authoritative Domains

Authoritative Domains are those for which an Exchange organization hosts mailboxes that have email addresses that use that domain.

For example, a company named Contoso Pty Ltd may own the domain name contoso.com and use email addresses of name@contoso.com.  The Exchange organization would be configured to consider contoso.com an Authoritative Domain.

An organization can have more than one Authoritative Domain configured.  Using Contoso Pty Ltd as an example again, they may have a second brand name of Contoso Services and use the contososervices.com domain name in marketing materials.  In this case the Exchange organization would be configured with both contoso.com and contososervices.com as Authoritative Domains.

Internal Relay Domains

Internal Relay domains are those for which an Exchange organization hosts some, but not all of the mailboxes that use that domain.  This scenario is sometimes also referred to as a “shared SMTP namespace”.

Internal Relay domains are common when two companies have merged but are yet to consolidate their Exchange environment into a single organization.  When they have a need for consistent email addressing across both Exchange environments Internal Relay domains are the solution.

When an Accepted Domain is configured as Internal Relay it tells the Exchange organization to accept mail for that domain, but if no recipient in that organization has that email address then it looks to the list of Send Connectors to determine where to send it next.

For example, if Contoso Pty Ltd and Northwind Traders formed a new company Contoso Traders with a new domain name of contosotraders.com, then each existing Exchange organization is configured with two items to share the SMTP namespace:

• An Internal Relay domain of contosotraders.com
• A Send Connector for the namespace contosotraders.com that sends email for unknown recipients to the other Exchange organization

External Relay Domains

External Relay domains are those for which an Exchange organization will accept email, but hosts no mailboxes for that domain.  This scenario might occur when one organization is acting as an ISP for other organizations, or offering services such as email content filtering.

External Relay domains are used when one Exchange organization is accepting email from the internet for a non-authoritative domain name, and then forwarding it on to the authoritative Exchange organization.  This is usually performed at the Edge Transport Server to keep email for non-authoritative domains from entering the corporate network.  For this to occur the Edge Transport Server is configured with two items:

• An External Relay domain
• A Send Connector for the namespace that sends the emails to the authoritative Exchange organization

In these scenarios it is also common for the Edge Transport Server to be used as an outbound email relay, or smart host, for the authoritative Exchange organization.

Summary

For most Exchange organizations the Authoritative Domain type is the only one used, however it is important for email administrators to understand the full capabilities of Accepted Domains as explained above.

Understanding Exchange Server Accepted Domains

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

In Part 1 of this series I discussed backup and recovery for each of the Exchange Server 2007 server roles and associated systems such as Active Directory.  In Part 2 I then described the process of backing up and recovering a Mailbox Server from the loss of a mailbox database.  In this part of the series I will demonstrate backup and recovery of the Hub Transport Server and Edge Transport Server roles.

Backing up Transport Servers

Unlike Mailbox Servers, the Hub Transport and Edge Transport roles do not require any special Exchange-aware backup software.  All of the necessary data for recovering a Transport server is contained within:

• Active Directory (for Hub Transport servers, but not Edge Transport servers)
• The Active Directory Application Mode (ADAM) database (for Edge Transport servers)
• The server’s file system
• The server’s System State

Hub Transport servers can be backed up using the built in Backup utility in Windows Server.  At the very least the backup should include the System State and the C:\Program Files\Microsoft\Exchange Server\TransportRoles location of the file system (and all sub directories).

Edge Transport servers are backed up in the same way as Hub Transport servers except for the Exchange Server configuration.  Because this is stored in ADAM it must first be cloned using the Export-EdgeConfig.ps1 script located in C:\Program Files\Microsoft\Exchange Server\Scripts.  Execute the script with the name of the file you wish to export to. Note this is a single command run on one line in the Exchange Management Shell.

export-edgeconfig.ps1 c:\edgeconfig.xml
 -key "abcdefghijklmnop"

It is recommended to either include this config file in your Edge Transport backups or use a path that is a shared folder on a remote server.

Recovering Hub Transport Servers

In this scenario the EXCHHUB server has been lost due to hardware failure.  Spare server hardware has been used to reinstall Windows Server 2003 along with the Exchange Server 2007 pre-requisites.  The newly built server has the same name and IP address of EXCHHUB.  Now we can begin the recovery of the Hub Transport server.
First, remove any Edge Subscriptions that existed for the Hub Transport server being recovered.  If you skip this step you may receive a certificate error during the recovery install.

In a command prompt run the following command from the location of the Exchange setup files.

setup /m:RecoverServer /DoNotStartTransport

This runs setup in recovery mode along with an additional instruction to not start the Transport services straight away.  This is so we can restore our mail queue databases and log files from the most recent backup before the server is put back into operation.

Recovering Edge Transport Servers

In this scenario the EXCHEDGE server has been lost due to hardware failure.  As with the Hub Transport server a new server and operating system has been installed with the same name and IP address.

To recover the Edge Transport server we will use the ImportEdgeConfig.ps1 script that ships with Exchange Server 2007.  First we run the script against the exported Edge config file from earlier to validate its contents. Note, this command is all one line.

ImportEdgeConfig.ps1 -cloneConfigData c:\edgeconfig.xml
 -isImport $false -CloneConfigAnswer c:\configanswer.xml
 -key "abcdefghijklmnop"

Validation succeeded for ConnectivityLogPath element of
 type DirectoryPath
Validation succeeded for MessageTrackingLogPath element of
 type DirectoryPath
Validation succeeded for PickupDirectoryPath element of
 type DirectoryPath
Validation succeeded for PipelineTracingPath element of
 type DirectoryPath
Validation succeeded for ReceiveProtocolLogPath element
 of type DirectoryPath
Validation succeeded for ReplayDirectoryPath element of
 type DirectoryPath
Validation succeeded for RoutingTableLogPath element of
 type DirectoryPath
Validation succeeded for RootDropDirectoryPath element of
 type NullableDirectoryPath
Validation succeeded for SendProtocolLogPath element of
 type DirectoryPath
Validation succeeded for SourceIPAddress element of
 type IPAddress
Validation succeeded for SourceIPAddress element of
 type IPAddress
Validation succeeded for Bindings element of type Bindings
Validation succeeded for Fqdn element of type FQDN
Answer File is successfully created: c:\configanswer.xml

Note that the key is the same one used when the Edge config was exported earlier.  If any validation steps were unsuccessful open the answer file in a text editor and adjust the settings that it lists as invalid.

Once the config has been validated it can be imported with the same command but modifying the -isImport parameter to $true. Note again this is a single command.

ImportEdgeConfig.ps1 -cloneConfigData c:\edgeconfig.xml
 -isImport $true -cloneConfigAnswer C:\configanswer.xml
 -key "abcdefghijklmnop"

Importing Edge configuration information Succeeded.

Now that the Edge Transport server configuration has been restored you can re-create the Edge subscription between the Edge Transport and Hub Transport servers.

How to Back Up and Restore Client Access Servers

In the final part of this series I will demonstrate the process of backing up Client Access servers and then recovering them from failure.

Exchange Server 2007 Backup and Recovery Part 3 – Transport Servers

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators