One of the great benefits for us in IT is that as hardware and storage prices have come down, and performance has increased, we are more able to offer services that in the past was prohibitively expensive to deliver. Rapid deployment and expansion of service, redundancy, and very high availability are all possible now for a fraction of the cost of a few years ago. Granted, it still costs more to provide such high quality service. Let’s take a look at how virtualization, replication and high availability, impact storage requirements and costs.

Virtualization allows us to deploy servers without tying resources to a single specific hardware system. The images can be moved from one system to another, cloned, made redundant and thereby easily allow expansion of particular applications and services. Virtual servers are a foundation for simple, rapid, consistent scalability. Having several or many identical instances allows us to deliver high availability far more easily. Virtual images do take space, and must run on a base platform, so clearly a single VM takes more space and resources that the same service running on dedicated hardware.

High Availability (HA) is the IT goal of having continuously available service for a particular application, connection or resource. Sometimes this is done via fail-over from a primary to a secondary connection or resource. It is also possible via load balancing. The load balancing can be accomplished at the application layer, at a gateway layer, or via an appliance. Load balancing is also possible at the name lookup level. For the purposes of this discussion we are considering application, gateway, and appliance types of load balancing and fail-over. Application layer mail gateway routing is often built-in to the system, whereby the gateway has alternate choices to try if its primary gateway is unavailable. This may be implemented in different ways depending on the vendor and the service. For SMTP there are underlying standards and requirements for gateway and routing behavior.

Replication of data that changes is key for us to have consistent service in the event of a failure of one of the data storage servers. So virtual images aren’t enough–we need to have the changing data replicated from the primary location to one or more redundant locations, ideally in real time.
Storage requirements obviously go up linearly for every replicated server. If you have a series of servers with the same OS, configuration, applications and local data replicated, you should then have for every n servers you have n times the storage requirements. For three servers, you have three times the base storage requirements. For 10 servers, 10 times the storage needed. Fortunately performance and reliability scale far better than the required storage. Another important factor is that the front end application layer or Web layer doesn’t hold all of the data presented. It should be clear that not every server hosts the directory of email addresses and user identities. And the Web interface doesn’t host the mail messages or the directory–it’s just a front end. The mail messages themselves are in a database, data store, or file store (depending on the mail server, platform and configuration you have picked) and that database can be highly available and replicated, but there isn’t a message store duplicating all messages on every server and replicating them to each one. Instead, the design is usually a central store, perhaps with one replica in a cluster. Similarly there are few directories, often replicated between sites or across long distances to improve performance for lookups by local users.

A very straightforward HA server layout might look something like this: two (or more) servers on the Web tier, two for each of the apps on the application tier, two directory servers and two message store servers for each site. So what might be possible to run on even one server or two in total, we now have eight servers defined, such that we have redundancy at every tier.

HA Cluster Architecture

We then  need to consider base storage requirements for each type of server along with the number of servers we are going to have of each type, in order to determine how much virtual drive space and/or SAN space our servers will consume. As we discover, we always want to budget toward the high end of space calculations, and then put in even more in our estimate for future unexpected situations. For example, on some of the servers we may want or need to take a snapshot of the entire message store to work with, but need to create it locally; so however big in gigabytes our message store is, we’ll need at least that much more room locally to copy or restore such an image. Repair and optimization tools for data stores and databases also may need similarly large amounts of space to work creating temp files or new copies of the data. So, a 15 GB virtual drive might seem big to begin with, but if you build up an 8 GB data store on it, you’re “out of space” if you need to make a copy locally. Consider also if you need to restore from a backup and don’t want to delete the in-place store. Of course, often such work can be done on network drives, but again be warned that disk performance will be much higher locally.

Many of the clustering and distribution of services concepts are available within Microsoft Exchange and are integrated into the application suite, but it’s worth it to understand how these ideas work independent of just a single messaging platform like Exchange.

Virtualization, Replication, Storage and High Availability

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

In my last posts I discussed Exchange Server 2007 Single Copy Clusters, which is one of two clustering features available in Exchange.  In this post I will discuss the other type of Exchange Server 2007 clustering, Cluster Continuous Replication

What is Cluster Continuous Replication?

Cluster Continuous Replication (CCR) for Exchange Server 2007 would not be familiar to anyone who only has clustering experience with previous versions of Exchange.  In a CCR cluster two cluster server nodes connect to non-shared resources in an active/passive configuration.  Exchange storage group and mailbox database information is replicated between the active and passive node using asynchronous log shipping.  I explained asynchronous log shipping in the first part of this series on the basic concepts of Exchange Server 2007 high availability.  The same log shipping occurs in Local Continuous Replication.

The two CCR cluster nodes appear to other computers to be a single system, and when one server node fails the clustered resources are able to fail over to the other node and continue operation.

CCR provides high availability for the Mailbox Server by protecting it from the failure of either cluster server node, as well as protecting it from storage failure.  Because each cluster node is connected to its own non-shared storage, the failure of one does not impact the other.  This also means that CCR disk storage can consist of much cheaper hardware alternatives than an enterprise-grade SAN would cost, making Exchange high availability a reality for businesses with smaller budgets.

CCR is available in the Enterprise edition of Exchange Server 2007.  Because it uses an underlying Windows Server failover cluster, the servers that form part of the cluster must also run the Enterprise edition of Windows Server (either 2003 or 2008).

Like LCR a CCR cluster offers high availability as well as performance benefits.  Backup operations can be performed against the passive copy of the database, making it possible to back up the server at any time of day without impacting end user performance.

Another advantage of CCR over SCC is that it increases the maximum recommended mailbox database size from 100Gb to 200Gb, thanks to the additional reliability and recoverability of a CCR cluster.

How to Install a Single Copy Cluster

In this example two Windows Server 2003 Enterprise edition servers have been configured with the basic requirements of a failover cluster:

• A public network interface for normal network operations
• A private network interface for the cluster “heartbeat”
• Matching disk configurations

Because there is no shared disk storage the cluster is installed as a Majority Node Set.

A Majority Node Set cluster needs to be configured with a third node called the File Share Witness.  The FSW acts as a “tie breaker” in the event that one of the cluster nodes fails.  One of the best places to locate the File Share Witness is on the Hub Transport Server in that site.

One the failover cluster has been configured we can install Exchange Server 2007 on the first node.  Run setup as normal, and when selecting the roles to install choose “Active Clustered Mailbox Role” as the server role.  You will notice that a clustered mailbox server cannot co-exist with any other server role.

Choose “Cluster Continuous Replication” as the cluster type, give the cluster a name, and specify a path for the database files.

Finally, give the cluster an IP address.

Exchange setup will configure all of the required cluster groups and resources for the underlying failover cluster automatically.  Once the active node is installed repeat the setup process on the passive cluster node, this time choosing “Passive Clustered Mailbox Role”.

Managing a CCR Cluster

For most Exchange administrative tasks the clustered mailbox server will appear like any other mailbox server.  For example, the storage group will show up as with any other mailbox server, but will have a value of “Clustered” for the Replicated attribute.

[PS] C:\>Get-StorageGroup

Name                      Server            Replicated
----                      ------            ----------
First Storage Group       EXCHMAILBOX       None
Second Storage Group      EXCHMAILBOX       None
First Storage Group       exchcms           Clustered

As with Single Copy Clusters a CCR cluster should not be administered using Cluster Admin (cluadmin.exe).  Instead, Exchange cluster operations such as moving the cluster group to another node should be performed in the Exchange Management Shell.

[PS] C:\>Move-ClusteredMailboxServer

Supply values for the following parameters:
Identity: exchcms
TargetMachine: exchmailbox2
MoveComment: Testing

Confirm
Are you sure you want to perform this action?
Moving clustered mailbox server "exchcms" to
target node "exchmailbox2" with move
comment "Testing".

[Y] Yes  [A] Yes to All  [N] No  [L] No to All
[S] Suspend  [?] Help
(default is "Y"):y

Similar to Local Continuous Replication the replication health of the CCR cluster should be monitored.

[PS] C:\>Get-StorageGroupCopyStatus

Name                SummaryCopy CopyQueue ReplayQueue
                    Status      Length    Length
----                ----------- --------- -----------
First Storage Group Healthy     0         0

The copy and replay queues should remain at zero or very close to it under most circumstances.  An increasing number for either can indicate a problem with replication.

Standby Continuous Replication

In the final post in this series I will demonstrate Exchange Server 2007 Standby Continuous Replication.

Exchange Server 2007 High Availability Part 4 – Cluster Continuous Replication

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

In my recent posts I discussed the fundamentals of Exchange Server 2007 high availability and how to use Local Continuous Replication.  In this post I will demonstrate one of the two Exchange Server 2007 clustering methods, Single Copy Clusters.

What are Single Copy Clusters?

Single Copy Clusters (SCC) for Exchange Server 2007 is basically the same as clustering in previous versions of Exchange Server.  Two cluster server nodes connect to shared resources in an active/passive configuration.  The two servers appear to other computers to be a single system, and when one server node fails the clustered resources are able to fail over to the other node and continue operation.

SCC provides high availability for the Mailbox Server by protecting it from the failure of either cluster server node.  Typically the cluster will also consist of redundant networking (e.g. teamed network interfaces, multiple switches) and storage components (e.g. a SAN that is in itself a highly available system through redundant components).  The servers are also usually located in high quality data centers with redundant power and cooling.

SCC is available in the Enterprise edition of Exchange Server 2007.  Because it uses an underlying Windows Server failover cluster, the servers that form part of the cluster must also run the Enterprise edition of Windows Server (either 2003 or 2008).

Unlike LCR an SCC cluster offers high availability benefits but no performance benefits.  Because there is a single copy of each mailbox database within the cluster there is no opportunity to use a passive copy for backup operations.

How to Install a Single Copy Cluster

In this example two Windows Server 2003 Enterprise edition servers have been configured with the basic requirements of a failover cluster:

• A public network interface for normal network operations
• A private network interface for the cluster “heartbeat”
• Shared disk storage for the database, log files, and cluster quorum disk

Once the failover cluster has been configured we can install Exchange Server 2007 on the first node.  Run setup as normal, and when selecting the roles to install choose “Active Clustered Mailbox Role” as the server role.  You will notice that a clustered mailbox server cannot co-exist with any other server role.

Choose “Single Copy Cluster” as the cluster type, give the cluster a name, and specify a path for the database files.  The database path must be on a shared cluster disk.

Finally, give the cluster an IP address.

Exchange setup will configure all of the required cluster groups and resources for the underlying failover cluster automatically.  Once the active node is installed repeat the setup process on the passive cluster node, this time choosing “Passive Clustered Mailbox Role”.

Managing a Single Copy Cluster

For most Exchange administrative tasks the clustered mailbox server will appear like any other mailbox server.  For example, the storage group will show up as with any other mailbox server.

[PS] C:\>Get-StorageGroup

Name                      Server            Replicated
----                      ------            ----------

First Storage Group       EXCHMAILBOX       None
Second Storage Group      EXCHMAILBOX       None
First Storage Group       EXCHCMS           None

You can view the status of the CMS using the Exchange Management Shell.

[PS] C:\>Get-ClusteredMailboxServerStatus

Identity                        : EXCHCMS
ClusteredMailboxServerName      : EXCHCMS.contoso.com
State                           : Online
OperationalMachines             : {EXCHMAILBOX1 <Active,...
FailedResources                 : {}
OperationalReplicationHostNames : {exchmailbox1, exchmail...
FailedReplicationHostNames      : {}
InUseReplicationHostNames       : {exchmailbox1, exchmail...
IsValid                         : True
ObjectState                     : Unchanged

Although the CMS is configured as part of the Microsoft Cluster Service it should not be administered using Cluster Admin (cluadmin.exe).  Instead, Exchange cluster operations such as moving the cluster group to another node should be performed in the Exchange Management Shell.

[PS] C:\>Move-ClusteredMailboxServer
cmdlet Move-ClusteredMailboxServer at command pipeline
position 1

Supply values for the following parameters:
Identity: exchcms
TargetMachine: exchmailbox2
MoveComment: Testing

Confirm
Are you sure you want to perform this action?
Moving clustered mailbox server "exchcms" to target
node "exchmailbox2" with move comment "Testing".

[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend
[?] Help  (default is "Y"):y

Cluster Continuous Replication

In the next part of this series I will demonstrate Cluster Continuous Replication for Exchange Server 2007.

Exchange Server 2007 High Availability Part 3 – Single Copy Clusters

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Microsoft Exchange Server 2007 has several different Mailbox Server high availability features included with the product.  Each of the features is similar to the others in some ways but also very different.

In this post I will explain each of the high availability features and which types of scenarios they are suitable for.

What is High Availability?

High availability is a term used to describe the avoidance of unplanned downtime for a computer system through the implementation of hardware and/or software solutions.  Generally speaking a high availability solution will involve the elimination of and single points of failure in the system, often by duplicating or replicating components of the system so that if one fails the other is able to continue performing the role.

An example of downtime would be an email server that has suffered a hard disk crash and is unavailable to users who are then unable to send or receive email.  An example of a high availability solution in this case would be the use of a RAID volume to protect from single disk failures.Exchange Server 2007 contains several high availability features in the Mailbox Server role that can protect a system from multiple types of failure.  These features are a combination of database replication and server clustering technology.

What is Asynchronous Log Shipping?

Some of the features I am going to describe will include the term “asynchronous log shipping“.  “Asynchronous” means “not synchronised“, and “log shipping” refers to the copying of a transaction log file from one location to another where it is then replayed into a replica of the original database to keep it updated with the changes made to the source database.

Exchange Server 2007 writes transaction log files of 1Mb in size, meaning as each log file reaches 1Mb it is closed off and the next transaction log file is created.  Asynchronous log shipping occurs after a transaction log file is closed off and no longer in use as the active log file.

In essence, asynchronous log shipping is how Exchange Server 2007 database replication occurs.

What is Clustering?

A server cluster is two or more servers working together to perform a particular role so that it appears to be performed by a single system.  There are several different types of clustering commonly used.

Compute Clusters – this refers to the combination of processing power to perform tasks at a high speed than a single system is capable of.  A compute cluster usually involves a master node and several slave nodes.  The master node hands off computational tasks to the slaves and then receives the completed tasks back from them.  For example, many animated movies are created using computer graphics that are rendered by compute cluster farms, with individual frames of animation being processed by different slave nodes.

Load Balanced Clusters – this refers to the combination of several systems to act as a single system by distributing workload across all of the cluster nodes.  For example, a cluster of two web servers will load balance web page requests, so that approximately half of the requests are served by one web server and half by the other.  Very highly trafficked web sites that need to handle millions of visitors each month will operate on load balanced clusters.

High Availability Clusters – these clusters, also commonly known as Failover Clusters, provide high availability for servers by having redundant nodes that are able to take over serving requests if the active node should fail.  Exchange Server 2007 clustering makes use of failover clustering.

Local Continuous Replication

In the next part of this series I will demonstrate the Exchange Server 2007 Local Continuous Replication feature.

Exchange Server 2007 High Availability Part 1 – HA Fundamentals

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators