As tightly integrated as modern email systems such as Exchange are, with the advanced features of the operating system, the enterprise directory, and the client systems, even small patches, changes and upgrades have the potential to wreak havoc. Large changes can be Herculean undertakings. If proper preparation, deployment and testing procedures are not followed, disasters are quite possible.
So consider that a seemingly simple upgrade to your mail servers, or a client security patch could result in significant downtime and give you and your IT organization a “black eye” due to the failure. We don’t want that. So, how can we avoid it?
Well, you may be thinking, “I’ve got backups, and even snapshots and images of the systems I’m altering, so if things go wrong, we can roll-back to the previous state almost instantly.” That’s great, and I hope you do have good backups, and even better restore procedures in place for when things do go wrong, as they will sometimes.
But how do we ensure that we are successful, so that we don’t need to quickly restore to yesterday’s configuration? After all, if we can’t get the changes and improvements in place, that will start to look bad as well. The question is, how do you test your changes, improvements, and upgrades? How do you ensure that when you roll the changes out into your live production environment that things will work properly and as expected?
What we need is a complete test environment that fully mirrors (as much as possible) the functions and operations of the production environment. Does it need to be identical? No. But what we need is an environment that has all the components in place. This is often overlooked in budgeting and in planning. Why, I don’t know. How management can expect you to successfully upgrade and expand features and functions without complete testing is beyond me. Without testing it is essentially a trial-and-error operation.
Testing on individual systems just doesn’t cut it. And trying to replicate the interactions of a modern email infrastructure with just two systems, one as “the server” and another “the client” is unlikely to be even a close simulation of an enterprise network’s messaging environment. Now, if you do have just a single email server, that’s great. You can probably test with a single box and a test client PC then. If your business is that size, you’re in luck. I would ask you to consider though, what happens if that email server crashes? What are you doing next? Do you have a spare system ready? Do you have to install the OS, then restore from your backup? Does that work? Have you tested it? If not, you should.
For a more extensive messaging infrastructure, we ask: what’s the best means of creating a test environment? There are several obvious options. First, you could clone the entire production network. Either setting up separate systems on an isolated, independent network or deploying virtual machines in a separate network can work here. The cloning and imaging of the systems is a solid plan, and can leverage the methods you use for backups and disaster recovery. Knowing that you can take images of your production systems and redeploy them and have everything continue working is a very comforting bit of knowledge.
There may be some benefit from creating new systems on the test network and installing the systems and applications “from scratch”. Why you would choose this is up to you, there may be a need to capture some of the processes, or the configuration complexity may be great enough that it’s actually simpler to install than to go in and modify the configuration manually on a copy of a system. This may not seem likely, but consider that if an automated install process updates a system, database, or remote host name in 20 places during the install, you’ve possibly got to go in and make those 20 changes manually on your clone.
We do want the test environment to be as close as possible to the production environment. So, we have to decide, is a scaled-down version of the infrastructure close enough, or do we need to make it exactly the same, just not with the same domain and host names, and client connections? That seems extreme. If we have four servers with mailboxes that have no difference in configuration than the particular user mailboxes present, we can be pretty sure that there is no need to duplicate that–one server should be sufficient for testing. On the other hand, if user directories and content are in an environment with automatic replication, we don’t want to test changes on systems that don’t also have that replication in place. Don’t assume something is not relevant–if it’s a “moving part” in production, you want it in your test environment.