Last month, I posted an article titled “What’s a good mailbox size?” where I discussed many of the considerations an architect must take into account when sizing storage for a new email system. In that post, I also set up a survey where I asked readers to answer six short questions about what they think makes a good size for a mailbox, as well as what future plans they might have for system growth. As promised in last month’s post, I am sharing the results of the survey now.

While the total number of respondents was somewhat less than I hoped for, the quality of those responses from survey participants is greatly appreciated. No one skipped any of the first four questions; the last two were “if” type and should have been skipped if not applicable. Thanks to all those who took the time to share their answers. I will share each question and the responses by percentage below.

1. What email system are you running now?

The vast majority of respondents are running Exchange, with a smattering of Notes, but nothing else apparently. While the dominance of Exchange was no surprise, that not a single Groupwise or Sendmail admin responded makes me wonder whether those products are dying out, or if their admins simply are too busy to deal with surveys.

2. What is the standard mailbox size (in MB) for regular users?

Almost half of those responding give standard users mailboxes less than half a gig. Those results shocked me on their own, but wait until you read question four!

3. What is the standard mailbox size (in MB) for special users (IT, executive, HR, or any other group that needs a larger mailbox)?

It looks like half of the admins who responded draw no distinction between regular users and special users. How egalitarian of you. We do see an uptick in the sizes though, with the other half of you giving special users 5 GB or more of storage space.

4. Is your current mailbox size adequate for the majority of users?

Okay, this is the answer that floored me. Over 90% of respondents feel their mailbox sizes are adequate. Considering how many provision regular users with less than half a Gigabyte, that is not at all what I would have expected. If you are one of those folks, please let us know in the comments how you do it. Do you have an aggressive deletion policy, an archiving solution, or do you just prohibit attachments? Inquiring minds want to know.

5. If you are considering a larger mailbox size for regular users, how big would you like that to be?

This is another response that did not come out like I expected. No one is looking for huge mailboxes (>25 Gigabytes) and most look like they would be happy with 10 GB or less. Storage vendors may weep to see this.

6. If you are considering a larger mailbox size for special users, how big would you like that to be?

Now we see an expected distinction, where special users will get much larger mailboxes. The designers of Exchange 2010 are smiling that so many of us share their vision.

So to summarise, the vast majority of regular users’ mailboxes are either in the 512MB range, or in the 1 to 5 GB range. In about half the cases, special users get a bump to 5 GB or larger. Most of us are happy with our mailbox sizes, but if we were going to increase them, it’s only the special users who’d really see a big change, and no one seems to want to go above 25 GB for mail.

Once again I’d like to thank those of you who participated in the survey for your time and the information you shared. I know it will be very useful to me, and I hope that other regular readers of TheEmailAdmin can also benefit from this.

Are there any other issues you’d like to see surveyed and discussed here? Any questions you’d love to see asked in such a large forum? If there are, leave a comment and if I get enough good ones, I will create another survey for the readers so we can see just how everyone else is doing it.

As it Turns Out, a Good Mailbox Size Is…

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

The other day I came in on the tail end of a conversation involving PSTs. The thought was to archive some former employees’ email into PST files, store them on the network, and let HR access them as needed. I waited for my turn to contribute to the discussion, and led in with “sure, and then we can roast marsh mallows over the burning server.” To say the reaction was underwhelming just doesn’t quite paint the picture. The silence was deafening. It seems that not everyone understands that PSTs are evil. Today, I hope to help fix that.

In a post earlier this year, my colleague Paul Cunningham discussed some options for “What to do With Those PST Files.” His article gives you some great alternatives to storing mail in PSTs. What I would like to do in this article is build upon that by giving you some of the reasons why you want to move away from PSTs.

I have encountered many situations where a company seems to have adopted a policy of network stored PSTs as the alternative to larger mailboxes. Whether it was to reduce the size of the information store, or simply because Exchange was deployed with smaller mailboxes in mind, the PST seems to be the logical place to store email that doesn’t have to be available online. And since storing a PST on a local hard drive puts that data at risk, most folks tend to move those PSTs over to a network share, like their home directory, so that they get backed up each night. This is a case of “good idea, bad reality.” To understand why, let’s consider the humble little PST.

The PST has been around, largely unchanged, since Exchange 4. It was designed as a short term storage solution, or for mail pulled from an Exchange server by “Internet Mail Only” mode users. The PST file is accessed by Outlook using a file-access-driven method built for local storage. This loosely parallels some virtual disk formats, where the single file holds a significant number of discrete binaries that may need to be accessed randomly. Outlook’s file-access-driven I/O does not work across network connections, so when a PST file is stored on a network drive, the access requests must be encapsulated in serial RPC calls, sent across the network, and then processed by the file server.

There are some significant challenges to performance when doing this. First, these RPC calls are serial, meaning they are processed one at a time, and one must complete before the next can commence. Secondly, the fileserver can exhaust both paged and non-paged pool memory to handle these requests, and may become unresponsive to other requests, recording events 2021 and 2022. These two limitations can lead to significant network performance issues as more and more users begin to store PSTs on the same fileserver. First, the server’s NPP will start to become exhausted. Second, response times will increase to the point where the server appears to be non-responsive for noticeable periods of time.

The performance impact of this is so severe, that Microsoft has published a knowledge base article specifically stating that Personal folder files are unsupported over a LAN or over a WAN link. Consider the significance of that. You cannot find a KB that states running a Quake server on your SQL server is not supported. You cannot find a KB that states running a public facing website on your domain controller is not supported. You can’t even find a KB article stating that defenestrating your Vista machine is not supported (no matter how good an idea that might seem to be,) but there is a KB telling you directly NOT to use PSTs over the network; it’s that big a problem. If you consult that KB, you will see that it has been updated and applies to every version of operating system through 2008.

If you are already in the situation of having PSTs on the network, what can you do? With Exchange 2010, the initial performance limitations from earlier versions of Exchange are gone, so as Paul mentioned in his article, moving the data back to the Inbox is a good way to go. In addition to the indexing, archiving, and discovery benefits he mentioned, my favourite plus is that the email will be available to users when they use OWA. I’ve gotten several calls in years past from remote users who need to get to an email stored in some archive. Additionally, Microsoft opened up the format of the PST file last year, and the market is starting to see third party tools to migrate data. If you have no choice but to keep data on PSTs for now, consider moving them back to the client for real time access, and replicating them to a file share strictly for backup. As long as the access remains local to the client, the adverse impact to the fileserver is not there. This is still not a recommended approach for long term use, but it is far better than accessing PSTs over the network.

Even more reasons to stop using PSTs

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

You’re ready to deploy Exchange, but you’re not quite sure if you have the storage chops to accommodate the change. Wouldn’t it be nice if you could simulate the installation and uncover any hassles that might be waiting for you when you start the process for real? Such a nicety exists. It’s called Jetstress, and Microsoft recently released  a guide on using it with any version of Exchange you plan to implement.

The 49-page document prepared by Senior Consultant Neil Johnson explains how Jetstress works, how to plan and perform tests with the software and how to automate the process. Although the guide discusses determining if a storage setup can meet the demands of an Exchange installation, it doesn’t offer guidance on storage design for the software. If you’re interested in that aspect of the preparation process you might want to consult another Microsoft offering: Mailbox Server Storage Design.

During the design phase of an Exchange deployment, theoretical targets are set for handling expected input-output loads for the software’s databases. Without Jetstress, those targets would be untested until Exchange was installed. Jetstress can be used to test those targets without such an installation. That’s important because it lets you see how your installed storage and server infrastructure will behave during an actual deployment of Exchange. Actual behavior, as predicted by Jetstress, can be very different from theoretical predictions cooked up during the design phase, especially when your storage infrastructure is shared or your storage design is complex.

Here are some Jetstress benefits cited in the guide.

• Validation that a physical deployment is capable of meeting specific performance requirements.
• Validation that a storage design can meet specific performance requirements.
• Identification of weak storage components before Exchange deployment.
• Proof of storage and I/O stability.

Something you need to take into account when using Jetstress–and which is stressed multiple times in the guide–is that the program needs to be run with the same version of the Extensible Storage Engine (ESE) files that will be used in your Exchange infrastructure when the software is actually installed.

A successful Jetstress test will validate all hardware and software components in the I/O stack from the operating system to physical disk drives. It will ensure that the components are working at a level that  meets the predicted performance needed for Exchange to operate successfully. You should keep in mind, however, that, like any test, Jetstress results are only as good as the data you give it, namely the user profile analysis and workload predictions formulated during the design phase of the project.

The Jetstress guide is organized into eight sections. After two introductory parts, it begins with a discussion of Jetstress internals then moves to when and where to run the program and how to install and configure it. It wraps up with advice on output files produced by the program and reading its report data.

Although you may be inclined to use Jetstress only during the build stage of an Exchange deployment, it can be useful during the design stage too, to take the guesswork out of that process. It can be used to give you some real numbers on the maximum storage I/O capacity of your system. Those numbers can then be used to create your bill of materials for the deployment more precisely, which can save you money on it and reduce risks associated with it.

Using Jetstress during the design phase, though, doesn’t preclude using it during the build phase. In fact, it’s still a good idea to perform the test again at that time. It’s common to test a single server and its storage solution at build time. That approach can miss problems such as someone installing only a partial part of the build on a server or an untested device driver lurking on a machine or existing faulty hardware. A Jetstress run can catch those problems, as well as identify failing components on the system.

“[I]t is much less stressful to identify a weak batch of disks during a Jetstress test than on a Monday morning after a large user migration!” the guide reminds administrators.

It also reminds administrators not to run the program on production servers with Exchange installed on them. That can lead to problems with Exchange performance counters.

The guide is chock full of useful information for using Jetstress and is worth a peek if you want to use the program to make an Exchange transition as smooth as possible.

Microsoft Jetstress guide useful for smooth Exchange transitions

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

With data burdens for organizations increasing at dizzying rates, storage management has become more important than ever. That wasn’t been lost on Microsoft in its continuing development of Exchange Server 2010. The software maker has applied a new philosophy to how the application handles storage. It takes into account the declining price of storage and the pressure to improve performance across the storage infrastructure. It embraces using direct-attached storage instead of disk arrays, continuous replication to spare servers instead of RAID or clustering and cheap disk arrays as a substitute for tape backups.

Exchange 2010, as did Exchange 2007, has improved its handling of input/output loads for a given number of simultaneous users. One way it does that is by shelving a technique for storing copies of email messages that Microsoft has used in all previous versions of Exchange. In those versions, Exchange tries to store all copies of a message at a single location on disk. That saves disk space but reduces performance. Exchange 2010 stores copies wherever there’s free space. That may eat up more space, but Microsoft felt the performance improvement was worth it. Those kinds of improvements in Exchange 2010 opens the door for IT departments to use more economical alternatives to traditional and expensive solutions, such as substituting serial attached storage for network attached storage.

While in economy mode, an IT department may be inclined to choose SATA rather than SAS drives. That may not be a wise idea in a high performance environment. SAS drives use a technology called Tagged Command Queuing (TCQ). It copies better under heavy loads than SATA drives, which use Native Command Queuing (NCQ). Both technologies improve performance by moving the management of I/O communication with a platter to the hard drive itself. The queue holding requests for hard drive action in NCQ devices, however, tops out at 31 requests. No such limitation is set on TCQ devices. In addition, SATA drives are built to a price point so they tend not to be as reliable as SAS drives. Moreover, the price differential between the two technologies isn’t significant when “peace of mind” considerations are factored in. According to Microsoft, SAS drives cost five percent more than SATA drives, yet perform 25 percent better under peak loads.

Deployment of RAID can be avoided, too, thanks to Standby Continuous Replication. It allows each server storing mail messages to be replicated on a hot-spare server. What’s more, the technology works over long distances so the hot-spare can be located at a distant location, which will make disaster recovery planners very happy. The approach can be very resilient. Microsoft, which has implemented the approach in its own operations, has found that over the three year life span of a server, an average of one drive will fail. When a drive fails, the database on that drive fails over to a replica server. According to Microsoft, its system experiences less than an hour of downtime per year.

Exchange 2010′s new storage technologies can also provide an escape hatch from total dependency on tape for backup storage. A major challenge of IT administrators is trying to cope with shrinking backup windows. As data grows, it takes longer and longer to back it up. Couple that with an expanding demand for availability due to business hours expanding from nine to five to 24/7 and some IT honchos can find themselves in a real bind. Of course, more hardware could be thrown at the shrinking backup window problem, but the kind of iron needed to meet past levels of performance can be expensive–both in terms of the kind of tape drives required and the media needed to feed the drives.

To some extent, backup problems are partially addressed by replication of data to the hot-spares. The difficulty there is there are no historic backups. If the backup on a hot-server gets corrupted, there are no earlier backups from which to restore the data. Microsoft’s answer to that was to create another level of nearline storage manned by SATA drives configured in a relatively inexpensive RAID 5 array.

The backup regimen for the array depends on an organization’s priorities. Microsoft, for instance, does an incremental backup every 15 minutes and a full backup overnight. That allows them to keep at least a week of backups on hand should something go wrong. Since even incremental backups can degrade system performance, the company makes its backups from the hot-spares. That diverts load demand from the live databases that are online and utilizes an under worked resource–the hot-spares.

Exchange 2010 has some storage twists

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

That is the question that came to my mind when I was considering the career options of Exchange Server administrators.

I know that other IT professions carry varying degrees of complexity, but still wonder how often the email admin is the smartest person in the room.  Putting aside the ego behind that question there are definitely a lot of areas in which an email admin needs to have an understanding.

Let’s consider some of the technical skills that a good email admin needs.

Email Servers – often the email administrator is working in environments with more than one email server product in production.  Even those who only manage one server product will still encounter other products as they deal with outside parties, often trying to troubleshoot a mail delivery problem.

Operating Systems – the email admin is also usually responsible for the operating system running on the server.  Again in heterogeneous environments this may mean several different editions of Microsoft Windows as well as some form of Linux or Unix.

High Availability – larger environments often require high availability for their email systems.  This means the email admin needs to understand cluster, network load balancing, and the Exchange Server high availability features.

Firewalls – every email system needs to move data to and from the internet, so an understanding of firewalls from different vendors is necessary.

DNS – this plays an important role in several ways, not only the MX records but also concepts such as split DNS and how important reverse DNS is for delivery.

Network Protocols – there are many protocols at different layers involved for an Exchange Server system such as TCP/IP, HTTP, SSL/TLS, RPC, and MAPI.

Email Clients – the email server admin will inevitably be called upon to assist with email client software support.  In my experience this extends not just to the different version of Outlook but also to POP/IMAP clients.

Mobile Devices – mobile access to email is so common these days that an email admin will encounter many different devices such as PDAs and smartphones requiring support.

Server Hardware – an email server is almost guaranteed to have the most concurrent users of any system in a business at any given time.  This means that the server hardware must be sized with enough resources to handle the peak load.

Storage – Storage Area Networks are everywhere these days and in my experience getting a well configured storage subsystem for an Exchange server requires the combined knowledge of both the storage admin and the email admin.

Backups/DR – in most businesses email is among the most critical systems and so it must be well protected by Exchange backups and able to be quickly recovered when disaster strikes.

Telephony – fax integration with email has been around for a while now and more recently telephone integration has also become mainstream.  Exchange Server 2007 introduced Unified Messaging that included many features previously only found in dedicated PBX systems.

Without a doubt being an email admin requires a broad skill set to manage what is one of the most complex and critical systems within a business.

Can you think of other important skills for email administrators?

Are Email Admins the Smartest People in the Room?

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Clark: "We hire attorneys for their IP and not their IT."

Electronic discovery has increased demands on storage systems, and that’s likely to continue.

According to Michael A. Clark, a managing director at EDDix LLC, an electronic discovery consulting firm, corporations with revenues greater than $1 billion is carrying around a caseload 150 active matters, 35 to 40 percent of which involve electronic discovery. With the new rules of Federal Civil Procedure adopted last December, he observed, “we’re going to see an ink blotting downward of electronic discovery to ever smaller matters.” He projects that within the next three years that 35 to 40 percent will move to 75 percent.

Finding information within the enterprise has always been a challenging task for legal ferrets, but those challenges have ballooned in recent times, according to Clark. “There are now not only more things to find, but more places to look for them than there had been before,” he said in a video interview posted at SearchStorage.com.

Finding information is a big challenge to operators of an enterprise network, but so too is deciding what should be stored and how long to store it, Clark noted.

“A number of corporations are devoting considerable resources to creating retention policies and then trying to enforce those policies,” he observed.

“You don’t want to be in the situation where you’re keeping everything forever,” he continued. “Corporations are telescoping the amount of time that they keep this stuff–not because it costs so much to store it, but a) it’s a liability and b) it costs an awful lot to go find it and retrieve it depending on what medium it’s on.”

When the discovery hammer is dropped on an IT department, it needs to be nimble enough to respond to the request, Clark said. “You need to preserve data when litigation is imminent,” he explained. “You need to be prepared to suspend all of the destruction processes that you have in place associated with the storage of that data.” Recycling of data tapes may have to be suspended, for instance, or “snapshots” taken of dynamic databases. In addition, the data should be in “litigation hold,” where it will be centralized and protected from alteration by users of the system.

Tools for collecting documents relevant to a litigation are very important to the discovery process, Clark said. The number one pain point from the general counsel’s point of view, he noted, is the ability to collect electronic documents. They must not only be collected from servers but from other places like desktops, laptops and other devices. For that kind of collection, there are programs that will create mirror images from desktops and laptops without disturbing the functions of those devices.

As the demands of electronic discovery increase, corporations, especially those in high litigation industries–financial services, energy, telecommunications, pharmaceuticals and tobacco–will begin to alter their infrastructures to accommodate those demands. Many of those companies, Clark said, are beginning to look at bringing some of the features and functions associated with electronic discovery–which heretofore may have been outsourced to a specialized electronic discovery services provider–inhouse so they can do some of those services themselves.

Among the services the companies will bring under their enterprise umbrella, he noted, are automated document collection and automated categorization of documents at the time of creation.

“So rather than looking at something forensically, after the fact, and then trying to figure out where it fits,” Clark explained, “we can add metadata to the document itself that would allow us to put it in buckets and more easily retrieve it and sort it downstream, with the presumption that all documents created are potentially evidence.”

In the storage area, he continued, businesses are looking more closely at near-line storage and archiving as an alternative to backup tapes. “Backup tapes were designed for disaster recovery and certainly were not designed and don’t function very well when we need to retrieve specific documents from specific custodians,” he argued.

He explained that the increased use of electronic discovery has raised the risk profile of corporations in litigation matters and pumped up compliance costs. Things like automated document collection and categorization, he continued, are being deployed in order to mitigate risk and cost.

Clark cautioned companies not to depend on attorneys for solutions to their electronic discovery problems. “Attorneys are not technologists and, by and large, not business people,” he explained. “What we’re talking about here is a business issue. It needs to be approached as a business process.”

“Attorneys have a tendency to be reactive,” he continued. “They tend not to be particularly proactive. Man-with-hair-on-fire is not a good way to live your life if you’re a corporation.”

“It’s the business people who are beginning to drive and should be driving many of the processes that represent best practices,” he added. “We hire attorneys for their IP and not their IT.”

E-discovery demands to double in three years

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

In-the-cloud storage got a setback last week when both HP’s and Yahoo’s online storage services were shut down for good. HP’s Upline has had a rocky past, with the young service having experienced numerous problems and delays, and reports of malfunction and inappropriate access. Of course, there is no shortage of other cloud vendors taking their place–and the ads are full of “Do you need an alternative to Upline” come-ons.

We’re still hearing rumors, and most recently, “confirmed rumors”, that Google’s Gdrive is ready to roll. Gdrive sounds revolutionary in design. It wil supposedly offer unlimited storage–allowing you to actually store the entire contents of your hard drive in the cloud. Local and online files are synchronized through a web interface so for example, you could start working on a project at the office, and then later on pick it up at an Internet cafe–or even on your smartphone. Gdrive will also be integrated with other Google applications and services. The security ramifications are immense though, and there’s as of yet no word as to how security would be provided. I’m not so sure I would want my entire hard drive replicated in the cloud. Also, it would seem doubtful that users who have to comply with various regulations regarding storage and backup would be able to take advantage of it, but we’ll wait to see on that one.

Before moving to an online backup environment, it would be important for a corporate user to review their data retention policy and any compliance requirements they face, and make a judgment as to whether in-the-cloud backup could adequately meet those policies and requirements.

How compliant is in-the-cloud storage?

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators