The Register published an article describing the removal of Single Instance Storage (SIS) from the Exchange Server 2010 database engine.  The comments on the article were largely critical of the change, declaring it a backward step in Exchange storage efficiency.

Before I discuss this further it helps to go back in time a little and understand what SIS is and how it came to be a part of Exchange Server.

Exchange Server Single Instance Storage

The essence of SIS is the storage of a single copy of data that is shared among different users or computers.  In the case of Exchange Server this meant a single copy of an email or file attachment is kept in the mailbox database, and then any user who received that item accesses it via a pointer to that one single instance of the item.

SIS was introduced in the Exchange Server product line in version 4.0 back in 1996.  Back then disk storage was very expensive compared to today’s prices.  Disks were much larger in physical size, smaller in capacity, and slower in performance.  Under those conditions SIS made a lot of sense to reduce the overall size of mailbox databases.

SIS remained a part of the Exchange database engine right up to Exchange Server 2007.  However in this time the price of disk storage has greatly reduced.  The database engine itself had also been improved and between Exchange Server 2003 and 2007 saw as much as a 70% decrease in IOPS (Inputs/Outputs per Second) requirements, meaning more users and larger databases could be stored on fewer disks.However some constraints still remained, such as a recommended maximum database size of 100Gb even with Exchange Server 2007.  This meant that many organizations deployed multiple databases within their environment, and of course a lot of organizations have multiple servers as well.

Because of this distribution of mailboxes across multiple databases and servers the benefits of SIS were beginning to decrease.  Remember that SIS is only effective within a single database.  If an email is sent to recipients on multiple databases each database stores its own instance of the data.

Instead of database reductions in the range of 20% the actual benefits were reducing to less than 10% in a lot of cases.  At the same time the Exchange Server product team were looking for ways to further improve the database engine.

Exchange Server 2010 Database Engine Improvements

These improvements were delivered in Exchange Server 2010, with further IOPS reductions of up to 50% and an increase in maximum database size to 2 Terabytes.  These improvements came thanks to a restructure of the database schema to reduce the number of tables that needed to be accessed by the average user session.  A side effect of this change is that SIS is no longer possible under the new table structure.

You might be thinking that the loss of SIS will cause Exchange databases to balloon in size.  Fortunately Microsoft has already addressed this by adding compression to attachments and header information stored in the database.  Microsoft says that this compression has been shown to be equally as effective, and in some cases more effective, than SIS at reducing overall database size.  Furthermore the processing overhead required to compress and decompress this data has been more than offset by the overall performance gains from the database schema optimizations.

More Efficient Server, Less Efficient Storage?

So what about data duplication then?  Many critics say that introducing data duplication into the Exchange database engine is a serious mistake and a step backward.  The trend in data architecture is towards data de-duplication.

These concerns are somewhat unwarranted though, for a few reasons:

• Data duplication is a problem that extends far beyond Exchange Servers.  Consider the average network drive full of multiple draft versions of documents.  The impact of Exchange on this, considering the compression gains, is negligible if any at all.
• The use of email as a file sharing medium is slowly reducing and this trend will continue.  Products such as SharePoint allow shared files to be referenced in email as a link instead of as a full attachment.
• Disk storage is very cheap, especially the SATA/SAS drives that are now completely capable of hosting the highly efficient Exchange Server 2010 databases.
• Where organizations choose instead to use more expensive SAN storage for Exchange Server 2010, de-duplication can occur at the block level as many SANs include this capability themselves.
• In a highly available Exchange Server 2010 environment data is going to be duplicated several times anyway thanks to the Database Availability Group feature.

Does the Removal of Single Instance Storage Mean Less Efficient Exchange Servers?

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

In my recent post I examined the question of whether Exchange Server 2010’s new replication and retention features could mean an organization no longer needed to back up their Exchange servers.

One reader emailed me to suggest that avoiding backups would be unwise, citing the example of potential corruption within the database that might replicate to all copies within the Database Availability Group and cause loss of data.

The scenario is one that definitely needs to be considered.  Although Exchange Server 2010 includes application intelligence to detect and handle some database corruption scenarios, there still exists the possibility of logical corruption leading to data loss.

Fortunately the solution to this problem is already included in the DAG feature of Exchange Server 2010.

If you are already familiar with Exchange Server 2007 Standby Continuous Replication then you already understand the concept of replay lag time.  Replay lag time is a configurable setting on each database copy within a DAG that delays the replay of a transaction log into that database replica.  A database copy with a replay lag time of more than zero is referred to as a lagged copy.

For example, within a DAG there may be 3 servers hosting replicas of Database 1.  Two of the servers are located in a primary data center, and the third server is located at a secondary data center.

The servers at the primary data center are configured with a replay lag time of zero, meaning the passive copy of the database is kept up to date with the latest transaction log information in near real-time.  A failure of one server permits seamless failover to the second server and continued email operation for the business.

The server at the secondary data center can be configured with a lag of 24 hours.  This means that even though transaction logs are immediately replicated to the server, they are not replayed into the database copy until 24 hours later.

This configuration means some additional configurations for the high availability design of the Exchange Server 2010 environment.

• The lagged copy is not for high availability through automatic failover, rather it is for disaster recovery scenarios.
• When a lagged copy is activated it can take some time to come online depending on the number of transaction logs that need to be replayed and the speed of the server.
• When a lagged copy is activated the administrator can choose to replay all log files, or only those log files up to a certain point in time.
• Because a lagged copy is generally activated only in a disaster scenario, it is possible that no other database copies within the DAG will be functional at that point in time.  To maintain a resilient Exchange Server environment either the server hosting the lagged copies should be configured with RAID storage, or more than one server should be configured with a lagged copy of the database.
• Lagged copies are themselves susceptible to corruption.  The servers must be monitored for event log errors that indicate a problem.  When a problem occurs the lagged copy must be reseeded, which therefore temporarily loses its lagged status.

What if You Never Backed Up Your Exchange Server Again? (Part 2)

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Imagine for a moment that you never had to back up your Exchange servers again.  Sounds crazy right?  Well with Exchange Server 2010, it may not be as crazy as you think.

In a recent post I described the killer new High Availability feature of Exchange Server 2010 – the Database Availability Group (or DAG for short).

A DAG is an Organization-level object that allows a database to have several passive replicas on other servers (DAG members).  When a DAG is configured it permits individual mailbox databases to failover to passive instances should any problem with the active database arise.

The nature of DAGs means that they can be deployed to protect from failures at almost any layer of the Exchange infrastructure.  DAGs can protect from anything from a single failed server hard disk to an entire data center failure as long as the DAG is architected accordingly.

How this relates to backups is simply this – if a database is protected from all failure scenarios by the DAG, why would you need to back it up at all?  Let’s take a closer look at this question.

Exchange Failure Scenarios

The DAG will protect data from any Exchange failure scenario:

Database Corruption –the database will simply failover to one of the passive copies.

Server or Server Component – all active databases on the server (or just those on the disk that failed if that is the case) will failover to one of the passive copies.

Network Failure – if a DAG member becomes unreachable another member will bring those databases online.

Site Failure – because DAGs can easily span multiple physical locations, even those in separate Active Directory sites, if a data center fails another DAG member at a different site will bring those databases online.

Other Data Loss Scenarios

Deleted Mailbox Items – Exchange Server 2010 introduces a new feature known as Single Item Recovery.  This means that when a user deleted a mailbox item it is put into a recoverable items area instead of being permanently deleted.  This area has a configurable retention period, and so the administrator can simply configure a reasonable period to allow people to recover deleted items without relying on backups.

Deleted Mailboxes – although it is quite easy to export mailboxes before deleting them this requires manual effort to do so.  However because of the Journaling capability of Exchange Server the mailbox items for deleted mailboxes can be recovered from the journal store.  In addition to this, deleted mailboxes are not immediately purged from the database and so can be easily reattached if necessary.

What’s the Catch Then?

You might be thinking from the above that it is entirely possible and practical to never back up your Exchange Server 2010 environment provided the right features are configured for replication and retention.

In fact in some organizations this level of protection will be quite acceptable.  But none of the above protects the Exchange environment from accidental or malicious deletion of Exchange Server data.

For example, if someone deleted the Database Availability Group all of the associated databases would also be deleted.  Without a backup these could not then be restored.

To protect from both accidental and malicious destruction of the Exchange environment you must make use of the new Role Based Access Control (RBAC) features in Exchange Server 2010.  Although RBAC itself is not a new concept, previous versions of Exchange Server never included the depth and granularity of control over access permissions that Exchange Server 2010 does.

Using RBAC to allow people only the minimum permissions they need to do their job can protect the environment from accidental or malicious deletion of data.  Ultimately though someone needs to be trusted with the highest level of permissions, which still opens the door to some risks.

It is likely that even the least risk adverse organizations will still employ backups of some kind for their Exchange Server 2010 environment, though perhaps on a less frequent schedule.

Do you plan to modify your normal backup practices in light of the new Exchange Server 2010 features?

What if You Never Backed Up Your Exchange Server Again?

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

I recently wrote a five part series about Exchange Server 2007 high availability, describing each of the HA features (LCR, SCC, CCR, and SCR) in detail and demonstrating their uses, strengths and weaknesses.

With the release of Exchange Server 2010 these high availability features are no longer available.  The features have now been replaced by a single high availability option known as Database Availability Groups (or DAGs for short, admittedly a horrible acronym for what is truly the product’s finest feature).

A little background is required to fully understand the evolution of Exchange storage architecture to its current incarnation in DAGs.  Going back to Exchange Server 2003 the product was limited to a maximum of 4 Storage Groups, each containing a maximum of 5 databases.  This meant a maximum of 20 databases (of either Mailbox or Public Folder type) per Exchange server.

Exchange Server 2007 improved on this in two ways.  Firstly, the product was limited to 5 Storage Groups and 5 databases for the Standard Edition, and up to 50 Storage Groups and databases for the Enterprise Edition.  The best practice with Exchange Server 2007 was to use one Storage Group per database, unlike Exchange Server 2003.

Naturally this lead to the question of why have Storage Groups at all?  Exchange Server 2010 answers this question by doing away with the concept of Storage Groups entirely.  Databases are now Organization-level objects that contain both the transaction logs and the database itself.

Exchange Server 2010 then introduces the Database Availability Group to provide high availability.  A DAG is a group of Mailbox Servers (up to 16 per DAG) that replicate databases and perform automatic failover and recovery for either individual database or entire server failures.

So how does this compare to Exchange Server 2007 high availability?  An immediate advantage is that DAGs unify Exchange high availability into a single feature and management toolset, unlike Exchange Server 2007 which had 4 different features each managed with different tools.

Let’s take a more detailed look at each of the features to see how they compare to DAGs.Firstly, Local Continuous Replication (LCR) was ultimately a pretty useless feature.  LCR protected from failure of a single database by replicating it to other local disk storage.  Most servers already use some form of RAID or highly available storage such as SAN, so protecting from storage failures was usually not necessary.  LCR was also cumbersome to activate in a failure scenario.  In comparison, DAGs offer more useful replication of databases to an entirely different server which can also be located at a different site, offering both host and site resilience.  The failover is also automatic.

The two cluster models, Single Copy Cluster (SCC) and Cluster Continuous Replication (CCR) also carried some limitations.  Failover occurred at the server level instead of the database level, so the failure of a single database for any reason caused all of the databases to failover to the passive cluster node.  DAGs on the other hand offer database-level failover.  Exchange Server 2007 also required that an underlying failover cluster be configured first.  Exchange Server 2010 configures the underlying cluster automatically when the first server is added to a DAG, allowing what is called incremental deployment (ie, you can make any Mailbox Server a member of a DAG without having to rebuild it as a cluster first).

Standby Continuous Replication (SCR), which did offer site resilience for Exchange Server 2007, was cumbersome to deploy, manage, and activate in the event of a disaster.  Failover to the passive replica was not automatic and required manual intervention.  It also required manual modification of user mailbox attributes using scripts.  Again DAGs provided automated failover, but in addition to this the redirection of user Outlook clients occurs automatically.

Finally one of the most attractive features of Exchange Server 2010 high availability is that it does not require that other server roles be located to separate servers.  Exchange Server 2007 cluster nodes were Mailbox Servers only and could not also be Client Access or Hub Transport servers.  Exchange Server 2010 allows these roles to stay on the Mailbox Server even when it becomes a member of a DAG.  This greatly lowers the hardware and software costs of deploying a highly available Exchange environment.

Overall the Database Availability Group is one of the killer features of Exchange Server 2010 and is sure to be a popular deployment option for Exchange Server users.

Exchange Server 2010 Delivers Improved Availability

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators