With more focus being placed on rapid recovery times for disaster recovery (DR) operations, much of the design, strategy, and practice work done for DR in the past has shifted more toward the high availability (HA) concept. For many businesses, an “always on, 24/7/365″ concept is key, so a recovery time of 48 hours is simply too long, and a data loss of an entire week would be catastrophic and considered a definite disaster in its own right. So, availability is now king–how do we achieve it? See my article on Virtualization, Replication, Storage and High Availability for introductory concepts on replication and how storage requirements increase, and on the general ideas behind clusters and replication.

Many of you here are from a Microsoft Exchange and therefore a Windows Server environment. While much has changed in the capabilities for Windows server clustering, especially in the Exchange area, many of the core concepts are the same regardless of what the latest features and options are. For example, block-level replication across drives on a SAN solution such as EMC’s SRDF/CE option is specifically designed to assist in replication of Windows databases such as SQL and Exchange, but the block-level replication works in essentially the same manner as DRBD does on Linux.

Clustering conceptually is the same regardless of the platform or systems as well. Although that might seem to be heresy to those that are irrationally tied to one platform or the other, it’s true. It’s even more true for dealing with the considerations for multi-site clusters or geo-clusters. Round trip times and network latency limits tied to the speed of light for geographically distant systems can’t be ignored, regardless of the platform or application. Also, clustering solutions have to deal with defining fail-over and fail-back procedures, and the theory behind most of these solutions is the same. Nodes in a cluster communicate via a heartbeat, and there is often a tie-breaker or “witness” node present to assist in validating that the primary node in the cluster has failed. For multi-site or geo-clusters, this is especially important both in the design stage and in understanding the possible failure modes. If network communication is down between sites, but not to and from clients at a site, multi-site clusters may fail-over and present a “split brain” situation where each site’s believes it is the active one, that the other is down.

Does the likelihood of a network outage mean that we must change our expected recovery time to be greater than the acceptable down-time for the network listed in our network SLA? Probably? This is a key question. How long must communication between sites be down before the secondary site decides that the primary site is really down and takes over as active?  Do you believe that having alternate paths for the heartbeat connection will solve this? Could that create an even greater problem? Let’s look at it:

Multi-path Communication for Multi-site Clusters
The servers will likely have a subnet spanning (cross-site VLAN) solution where their heartbeat network interfaces communicate. This network path therefore includes distinct network adapters (NICs), cabling, possibly separate switching, and may take a different path to and from the remote site. If the sites communicate via a traditional WAN link, but clients connect between sites or to each site via separate Internet facing routers or VPN concentrators, the client path to the remote site and its server(s) in the cluster may be very different. Consider already that client communication on the primary site with the active node(s) may fail, but the different network path for the hearbeat and quorum info may have the cluster in a state where it is healthy, but unreachable.

If the cluster fails over due to heartbeat communications failing, but when clients can still reach the primary site’s active servers, very strange problems can arise. Depending on how DNS is configured, and on how the cluster’s IP address is managed, clients might be directed to the secondary site based on the interruption of communications on the heartbeat network. In fact, the primary site is still active. Depending on the SAN or replication solution, one or the other of the sites will be writable with the data, while the other is just being replicated to. The load-balancing or DNS management needs to align with which cluster site is active. If the heartbeat network goes down and the cluster fails over to the secondary site, but clients are still directed to the primary site by a load balancer or DNS, that site likely won’t have access to the disk volumes since the SAN will have failed over to the secondary. If the replication solution still allows write access, the data between sites will be inconsistent. The cluster will think the secondary site is active, yet data has been written to the primary. Granted, if things are set up correctly this should not happen. But it can. Be warned.

Considerations for High Availability Designs Used for Disaster Recovery

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

As companies and organizations grow in size, departments internally supporting the business grow as well. IT of course is one that must scale to accommodate business needs. If your department is small, it’s very likely that you know how all the components in your IT infrastructure are configured, what they are, what they do, and so forth. You know not only which servers host what resources, but know about the configuration of users in Active Directory, you may be responsible for provisioning those users, and for setting them up with VPN access, server access, and other actions unrelated to configuring the user in Exchange or giving them a mailbox as well as a login. You may be thinking, “Of course, Clemmer, but doesn’t everyone know about all the elements in a network and how the interrelate with email?”

Well, in larger organizations both operational responsibilities and security policies make the separation of duties for IT staff a reality. What does this mean? Well, the person who manages the firewalls and configures rules to allow email traffic between company sites or business units is very likely not the same email admin who is going to configure the SMTP connector or inter-site replication. The staff member that gets information from human resources and provisions accounts is likely not the same staff member that builds out hardware for servers, or configures desktops or notebooks for the new users. The security staff that manage proxies, load balancers, network anti-virus solutions and other security solutions are not the ones that will perform tuning and regular maintenance to your email servers, in most all cases. If you have backup and storage managed by a separate group in the IT staff, they may or may not know the specifics of backing up an Exchange database or server.

What will all the results of this separation of duties be? Will things work better or more poorly? Are you already in this sort of situation and frustrated that nothing seems to get done and that things take many times longer than they used to or seem that they should?

If you are a growing organization and thinking of separating duties and responsibilities because of workload, security, expertise, or all three, consider carefully what the impact will be. When one group does not know what another is doing, when, or why, it can make otherwise simple changes into boondoggles. Scheduling with clear communication between groups of planned outage times, priorities, and potential risks of course are important. Clear communication sounds easy, but when everyone is busy with their own work sometimes we forget that not everyone knows what we are doing and that everyone else may not have read every single email before they left work for the weekend, especially if the email was about another group’s project. When things do go wrong and problems erupt, affecting systems unexpectedly, is there a well-understood escalation process and means of contacting the staff needed to troubleshoot and resolve things? Monitoring systems and automatic email or text alerts are great as long as those systems can function properly and they have a connection outbound to the Internet and from there to you when the crisis happens.

Recently we discovered at one location that backups for some systems had not been running for a long time. No alerts or warning about that, because the backups weren’t configured in the first place. A few days later we discovered that some of the systems were not being monitored for performance at all, although there was monitoring software available and a plan was in place, it just wasn’t happening. These things went unnoticed by the staff directly administering and supporting those systems, because they did not have administrative or even read-only access to the backup technology or the monitoring solution. The staff did not have the means to even look and see if these important functions were active and operating as assumed. The problems have been corrected, and going forward, the staff has been granted access to check that the backups and monitoring are operating. This is an example of where separation of duties was problematic. The lessons learned were that we can’t assume that others know what we want, and that we should verify things. Just trusting someone in another area’s word that something is true isn’t enough–”show me” works better.

Organizations can be and will be so large that any one IT staffer simply can’t know everything about everything. The field is becoming complex enough that this is no longer possible. For large organizations, it’s not possible to have the same group manage every IT service. Since this is the reality, we are left with the task of ensuring that the different IT roles can and do work best together. As an email admin you may discover that you know more and more about less and less of the whole IT infrastructure. Just don’t take it to the point where you know everything about nothing!

Is Separation of Duties in IT a Help or a Hindrance?

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

As tightly integrated as modern email systems such as Exchange are, with the advanced features of the operating system, the enterprise directory, and the client systems, even small patches, changes and upgrades have the potential to wreak havoc. Large changes can be Herculean undertakings. If proper preparation, deployment and testing procedures are not followed, disasters are quite possible.

So consider that a seemingly simple upgrade to your mail servers, or a client security patch could result in significant downtime and give you and your IT organization a “black eye” due to the failure. We don’t want that. So, how can we avoid it?

Well, you may be thinking, “I’ve got backups, and even snapshots and images of the systems I’m altering, so if things go wrong, we can roll-back to the previous state almost instantly.” That’s great, and I hope you do have good backups, and even better restore procedures in place for when things do go wrong, as they will sometimes.

But how do we ensure that we are successful, so that we don’t need to quickly restore to yesterday’s configuration? After all, if we can’t get the changes and improvements in place, that will start to look bad as well. The question is, how do you test your changes, improvements, and upgrades? How do you ensure that when you roll the changes out into your live production environment that things will work properly and as expected?

What we need is a complete test environment that fully mirrors (as much as possible) the functions and operations of the production environment. Does it need to be identical? No. But what we need is an environment that has all the components in place. This is often overlooked in budgeting and in planning. Why, I don’t know. How management can expect you to successfully upgrade and expand features and functions without complete testing is beyond me. Without testing it is essentially a trial-and-error operation.

Testing on individual systems just doesn’t cut it. And trying to replicate the interactions of a modern email infrastructure with just two systems, one as “the server” and another “the client” is unlikely to be even a close simulation of an enterprise network’s messaging environment. Now, if you do have just a single email server, that’s great. You can probably test with a single box and a test client PC then. If your business is that size, you’re in luck. I would ask you to consider though, what happens if that email server crashes? What are you doing next? Do you have a spare system ready? Do you have to install the OS, then restore from your backup? Does that work? Have you tested it? If not, you should.

For a more extensive messaging infrastructure, we ask: what’s the best means of creating a test environment? There are several obvious options. First, you could clone the entire production network. Either setting up separate systems on an isolated, independent network or deploying virtual machines in a separate network can work here. The cloning and imaging of the systems is a solid plan, and can leverage the methods you use for backups and disaster recovery. Knowing that you can take images of your production systems and redeploy them and have everything continue working is a very comforting bit of knowledge.

There may be some benefit from creating new systems on the test network and installing the systems and applications “from scratch”. Why you would choose this is up to you, there may be a need to capture some of the processes, or the configuration complexity may be great enough that it’s actually simpler to install than to go in and modify the configuration manually on a copy of a system. This may not seem likely, but consider that if an automated install process updates a system, database, or remote host name in 20 places during the install, you’ve possibly got to go in and make those 20 changes manually on your clone.

We do want the test environment to be as close as possible to the production environment. So, we have to decide, is a scaled-down version of the infrastructure close enough, or do we need to make it exactly the same, just not with the same domain and host names, and client connections? That seems extreme. If we have four servers with mailboxes that have no difference in configuration than the particular user mailboxes present, we can be pretty sure that there is no need to duplicate that–one server should be sufficient for testing. On the other hand, if user directories and content are in an environment with automatic replication, we don’t want to test changes on systems that don’t also have that replication in place. Don’t assume something is not relevant–if it’s a “moving part” in production, you want it in your test environment.

The Importance of a Testing Environment

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

One of the great benefits for us in IT is that as hardware and storage prices have come down, and performance has increased, we are more able to offer services that in the past was prohibitively expensive to deliver. Rapid deployment and expansion of service, redundancy, and very high availability are all possible now for a fraction of the cost of a few years ago. Granted, it still costs more to provide such high quality service. Let’s take a look at how virtualization, replication and high availability, impact storage requirements and costs.

Virtualization allows us to deploy servers without tying resources to a single specific hardware system. The images can be moved from one system to another, cloned, made redundant and thereby easily allow expansion of particular applications and services. Virtual servers are a foundation for simple, rapid, consistent scalability. Having several or many identical instances allows us to deliver high availability far more easily. Virtual images do take space, and must run on a base platform, so clearly a single VM takes more space and resources that the same service running on dedicated hardware.

High Availability (HA) is the IT goal of having continuously available service for a particular application, connection or resource. Sometimes this is done via fail-over from a primary to a secondary connection or resource. It is also possible via load balancing. The load balancing can be accomplished at the application layer, at a gateway layer, or via an appliance. Load balancing is also possible at the name lookup level. For the purposes of this discussion we are considering application, gateway, and appliance types of load balancing and fail-over. Application layer mail gateway routing is often built-in to the system, whereby the gateway has alternate choices to try if its primary gateway is unavailable. This may be implemented in different ways depending on the vendor and the service. For SMTP there are underlying standards and requirements for gateway and routing behavior.

Replication of data that changes is key for us to have consistent service in the event of a failure of one of the data storage servers. So virtual images aren’t enough–we need to have the changing data replicated from the primary location to one or more redundant locations, ideally in real time.
Storage requirements obviously go up linearly for every replicated server. If you have a series of servers with the same OS, configuration, applications and local data replicated, you should then have for every n servers you have n times the storage requirements. For three servers, you have three times the base storage requirements. For 10 servers, 10 times the storage needed. Fortunately performance and reliability scale far better than the required storage. Another important factor is that the front end application layer or Web layer doesn’t hold all of the data presented. It should be clear that not every server hosts the directory of email addresses and user identities. And the Web interface doesn’t host the mail messages or the directory–it’s just a front end. The mail messages themselves are in a database, data store, or file store (depending on the mail server, platform and configuration you have picked) and that database can be highly available and replicated, but there isn’t a message store duplicating all messages on every server and replicating them to each one. Instead, the design is usually a central store, perhaps with one replica in a cluster. Similarly there are few directories, often replicated between sites or across long distances to improve performance for lookups by local users.

A very straightforward HA server layout might look something like this: two (or more) servers on the Web tier, two for each of the apps on the application tier, two directory servers and two message store servers for each site. So what might be possible to run on even one server or two in total, we now have eight servers defined, such that we have redundancy at every tier.

HA Cluster Architecture

We then  need to consider base storage requirements for each type of server along with the number of servers we are going to have of each type, in order to determine how much virtual drive space and/or SAN space our servers will consume. As we discover, we always want to budget toward the high end of space calculations, and then put in even more in our estimate for future unexpected situations. For example, on some of the servers we may want or need to take a snapshot of the entire message store to work with, but need to create it locally; so however big in gigabytes our message store is, we’ll need at least that much more room locally to copy or restore such an image. Repair and optimization tools for data stores and databases also may need similarly large amounts of space to work creating temp files or new copies of the data. So, a 15 GB virtual drive might seem big to begin with, but if you build up an 8 GB data store on it, you’re “out of space” if you need to make a copy locally. Consider also if you need to restore from a backup and don’t want to delete the in-place store. Of course, often such work can be done on network drives, but again be warned that disk performance will be much higher locally.

Many of the clustering and distribution of services concepts are available within Microsoft Exchange and are integrated into the application suite, but it’s worth it to understand how these ideas work independent of just a single messaging platform like Exchange.

Virtualization, Replication, Storage and High Availability

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Certificates and encryption utilizing them play a critical role in modern systems and network security. Even if none of your email users has a client certificate in their email application, and they’re not using PKI for a VPN connection, they’re using certificates in more than a couple of places on a Windows network with Active Directory and Microsoft Exchange. You say, “Clemmer, I know all this, so what?”

Certificate Import Wizard

As I discovered recently, the need to renew certificates only once every year, two years, or more, can make for some hair-pulling troubleshooting with turnover with IT departments often shorter than that time period and likely sparse internal documentation for the many “set it and forget it” configuration components of the CA infrastructure.

Managing certificates can be relatively easy or could be (or become) rather difficult depending on how you go about it and how far you leverage the tools available to assist or automate things. Lack of user understanding for this very technical topic, along with frequent confusion on the part of new administrators as well as complex and dry documentation can all contribute to problems. Another area of confusion is the many places that certificates can be integrated and the different roles of certificates in your infrastructure.

Your Active Directory domain is going to require Microsoft certificate services provided by at least one Microsoft Certification Authority running on a server, often a domain controller. You can configure a Certification Authority (CA) numerous ways, and they can have various roles. Best practices define a number of design specifics that are generally well documented in Microsoft’s training materials and on TechNet. An Enterprise CA requires Active Directory and can be used to “issue certificates for purposes such as digital signatures, secure e-mail using S/MIME (Secure Multipurpose Internet Mail Extensions), authentication to a secure Web server using Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and logging on to a Windows Server 2003 family domain using a smart card.” CAs can publish certificates to Active Directory (AD) for users and computers as well. A number of CA policy, certificate templates, domain security policy, and AD security settings must be set correctly for certificates to be published automatically.

Email and Microsoft Exchange specifically can leverage certificates in several ways, with internal transport certificates, self-signed certificates, SMTP TLS certificates, and more. Check out this Certificate Use in Exchange Server 2007 on the various uses of certificates in Exchange Server. Certificates are used to encrypt LDAP communication as well, although Exchange normally uses self-signed certificates to encrypt LDAP communication between its ADAM instance (at the Edge Transport) and internal Active Directory servers. Most email admins are aware of the use of certificates in Web SSL traffic, and with Exchange SSL is used to communicate between various Web clients and Client Access servers. Even ActiveSync users that connect to Exchange use SSL certificates to encrypt their sessions.

In almost all cases where the communication traverses an unsecured, partially secure, or untrusted partner network, you will likely want to use a third party, external X.509 CA. When communication is with internal resources, but perhaps ones that are remote, an internal enterprise CA may be preferable due to cost. This leads us back to some points made at the beginning of this post, that an internal CA is a CA that you must manage. Clients and computers will need certificates, and those certificates will expire after a year or two. Servers and applications will also be issued some of these certificates, and they too will need renewal when the time comes. You say “Clemmer, I’ll just increase the key size and set the certificate expiration to five years.” That may delay the renewal effort, certainly–but it won’t eliminate the effort–only delay it. Cracking private keys for certificates with large key sizes isn’t much of a concern today, but with the continuing increase in computing power who it to say that a standard sized private key can’t be cracked four years from now? Consider too that some keys will need to be re-created when new server services appear, and your choice when upgrading services, applications, and operating systems will be to either migrate the certificates you have, or regenerate all of them. Will you really be running the same systems in five years?

I found the solution to my curious problem with certificate renewal in our two-level domain hierarchy in this TechNet article. It’s worth a read if you are expanding your Windows network or designing an improved domain structure.

Microsoft Certification Authority, Certificates, Your AD forest, and More

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

My recent posts have discussed identifying commonalities in new occurrences of spam, and concerns to keep in mind regarding indirect attacks using email as a vector. A strong perimeter defense and solid virus protection, along with an effective anti-spam solution can lull us into a false sense of security. The seemingly constant stream of unwanted mail begins to look like little more than an annoyance and not a continuing threat. In this post let’s examine technically other methods of attack, how to recognize them, and ways and means to defend against them.

Attacks against email servers, systems, and infrastructure are in many ways similar to attacks against other Internet-facing services, but are different in several important ways. Just as a concerted attack that brings down your Web servers stops communication with customers, vendors, and others on the Internet, the same is true for email communication attacks.

Email is such a lifeblood of business communication that if it fails it could be considered a critical failure. If your business relies on email for critical types of communication and seldom has problems, it may not be immediately obvious to you or those trying to communicate with you that the communication channel has failed. Of course, if you receive lots of email and suddenly it stops coming, it may be obvious very quickly that there is a problem. You might not notice though, if internal mail is fine but Internet emails slow or stop. Let’s consider what sort of attacks specific to email we need to be aware of, as well as more generic attacks that can target any Internet-facing system.

1. Denial of Service (DoS) Attacks (total connections)  – If you have experience administering Internet systems you are likely aware of the threat of DoS attacks. Your firewall should have the ability to mitigate or prevent these sorts of attacks by slowing the rate or total number of connections inbound to your email servers. Some types of servers can handle more connections than others. For example, a Web server that serves up simple (mainly text/HTML) Web content that isn’t graphics or media-content heavy can handle many more connections than a server that streams media. Servers can be tuned to end connections sooner and other connection-increasing settings, as well, but the firewall is where your best protection resides. SMTP servers often can have long wait times or receive files that transmit slowly, so lasting connections are a known problem, as we’ll see in the next type of attack.
2. DoS (attachment content & size) – Related to the DoS attack based on overloading the server with connections, an SMTP server can be overloaded with transfers of large attachments or ones that upload so slowly that the connection lasts far longer than is reasonable. The transfer taxes resources on the server that will not be released until finished. Other attacks may involve transferring attachment files that contain payloads that crash anti-virus or content-scanning modules, or otherwise harm or crash attachement spooling or queues. Zipped attachments that are corrupt or maliciously modified are problematic as well.
3. Mailbox stuffing or rejection overload (server sends bounce/reject mail/NDR reports in Exchange) – Some attacks intentionally overload or “stuff” particular mailboxes that are known to exist or are discovered to be valid. If the spam filtering does not block the messages, a massive, overwhelming number will be present and can crash the server or the mail client. A related type of attack is to forge the sending headers and attempt to send mail that will fail, generating a bounced (or rejected) email report message, such as an NDR report in Microsoft Exchange. An unwitting victim is subject to a flood of failure emails for mail they did not send. Modern mail servers are less likely to fall prey to this sort of manipulation, but it is sometimes possible.
4. SMTP Auth Attacks (Exchange) – Some mail servers are used by remote users, and to prevent unauthorized users from relaying mail via the SMTP server, it is set up to require authentication. SMTP authentication is only as strong as the password used. A weak password on a known account can make the SMTP server as exposed as an open relay–with a false sense of security. A better method for allowing remote users to send mail is either a strongly secured interface, perhaps a Web mail interface requiring strong authentication such as client certificates or USB tokens, or better yet access to email only via VPN.

Hopefully you won’t run into the worst of any of these sorts of attacks. Before you do, consider your options to prevent them from harming your email connectivity. If you don’t have a strong firewall with good defense against DoS attacks, you really should get one. Email security monitoring and alerting solutions can help detect the other types of attack before they get out of hand.

Email Attacks and Defense Against Them

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Sometimes spam, viruses, and other malware filtering at your email gateway isn’t enough. It’s important to keep your host anti-virus signatures up to date, and if you don’t have anti-virus protection at your firewall or on your network at the Internet gateway you should seriously consider it.

Here’s why these items are critical. Some recent malware attacks have used malware embedded in video and audio streams as a transfer. They can gain an initial foothold, so to speak, by managing to get a link to your users in a spam email. If your spam filter doesn’t block the message, a link in the email appears to be a video or audio link, but in fact the destination contains a trojan that is embedded in the content stream.

This method of attack isn’t exactly new. For example, the ZLOB Trojan began making rounds in 2005, and began gaining traction in 2006. Some attacks with it simply involved downloading other viruses or malware. Using a video link, however, for users that have their ActiveX controls set to download codecs automatically means that those users with poor virus protection would automatically download the virus and become infected.

Now, most of us won’t have this problem, right? Surely you and your users would, at a minimum:

1. Have host-based as well as network/perimeter-based anti-virus protection.
2. Keep your anti-virus signatures up-to-date for all your systems.
3. Not have your browsers set to automatically download and install ActiveX controls or codecs.
4. Have users trained, understanding not to install random codecs or ActiveX controls themselves.
5. Have in place strong anti-spam protection that may block messages from domains likely to send these messages.
6. Have perimeter security measures in place that detect and block or intercept malicious content as it appears.
7. Have users trained well on the risks of clicking unknown links, or going in search of suspicious content.
8. Have a proxy or firewall with content filtering in place, with a policy that prohibits visiting or traffic from certain domains known to be sources of malware.
9. Keep your systems patched with the latest security patches from your OS vendor and from your application vendors.
10. Frequently review your security protections and rules in place, and carefully consider before making changes allowing more permissive use and access to and from protected resources.

The most security conscious of us and those that keep current with security risks and trends in security technology may think that all of this is old news, that of course they won’t have any problems–and they may be right. I hope so. However, new small businesses and new business Internet users are appearing all the time. As these businesses grow and expand, they may have transition periods where their deployed technology changes and of course upgrades will happen sometime. At those times, extra vigilance is required. If you are brought on board during a transition period as an email administrator, network administrator or security administrator, be aware that such risks are heightened.

While the attempt to execute malicious code via a codec installation may seem to be old hat, consider that new vulnerabilities appear frequently. Consider that Windows Media Player can play streaming content, and couple that with the recent vulnerability MS09-047, Microsoft Windows Media Playback Memory Corruption Vulnerability. This vulnerability can permit remote code execution. Exactly the sort of vector needed by the sender of the spam we started this discussion with. A maliciously crafted Windows Media Format file pointed to by a link in a spam email. Granted, this vulnerability and other like it have been patched, and if you are up-to-date on your patches it isn’t actually a threat.

Where this can become a problem (and as far as I know it isn’t with this vulnerability) is when the patches interfere or conflict with mission critical applications and can’t be applied, and when system updates (unfortunately including some antivirus and security patches) that may require reboots can’t be done as soon as they are received. Testing and verification may be required in your business (and is a good idea if it’s not part of your routine) before applying new patches and updates. During this window of time, when the attacks are launched on “zero day”, till your patches are applied, your systems may be vulnerable. During this (hopefully brief) time period the sort of attack described at the beginning of this post could actually penetrate your security and wreak havoc. Follow the ten tips listed above, and minimize your vulnerability.

Malware Threats from Unexpected Sources: Trojans Embedded in Streaming Video Links

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators

Despite the generally excellent performance of most modern, well-tuned anti-spam engines, some spam is going to get through. We may be lulled into a false sense of superiority when for a period of time our anti-spam tools and techniques have borne fruit, and we see that we have more-than-just-excellent results; we have no spam in our inboxes for an entire day, week, whatever. Then, it returns. We’ve all seen it happen. Some strangely formatted message that you or I can surely tell is garbage, a bizarre attempt to sneak through your heuristics that has surprisingly succeeded.

Lately it has been some rather clever nonsense. I’ve been getting these spam emails with a particularly peculiar twist. Many of them have what appear to be at first glance meaningful, but “non-spam” sentences. On closer look, the sentences are strange, and not quite sensible. For some reason they consistently were getting through the spam filtering. What was strangest to me was the lack of any marketing content or attempt to sell whatsoever. They did have a link in the message, and the link was not ever to the same web destination or even clearly directed to an obvious undesirable site. This may have been one of the reasons this set of spam got by; to the filters, it looked really no different than a sentence or two sent by a friend describing some link they thought I would be interested in.

The content appears to be randomly generated by some sort of sentence constructor, which picks nouns, verbs, adjectives and strings them together, so that they seem to be part of a coherent sentence, but are not. The sentences are not riddled with attempts at sales or exciting your interest; instead they are just random. Oftentimes eerie in their close-but-not-quite structure.  Here’s an example, to show what I mean.

Part of him was shocked, but most. of him wasn’t even surprised. seen that right away.
There were maybe fifty in all, most. no bigger than plump raisins. No.

This is just one of the most recent ones. Often they have better punctuation, notice that this one has a few periods without spaces following and missing a few capital letters. One thing we don’t see is the crazy mixed-case words, with sexual content misspelled intentionally and with an obvious attempt to excite or lead us on into clicking the link that was attached and apparently unrelated to the text.

Now here’s the thing I found problematic. I can’t see where this content is going to work to be parsed in an anti-spam scanner in most cases, as it’s random enough when compared with the other spam of the same “type”, and yet the content could easily be valid if you wrote me: “Part of him was shocked, but most of him wasn’t even surprised.” Does it make sense to try to include this in our heuristic anti-spam scanners? I think not. We have to combat this by another means.

An old standby would have been to block inbound messages from this sender or IP address, but unfortunately this one came from Hotmail and I just can’t see blocking all email from any Hotmail senders, as much as I might want to do it some days. That was the first thing to do, though, is examine the headers and the log files to be sure that the mail did in fact come from where it claimed, from a Hotmail address and not from some other source. I still see significant forging of email headers.

The next comparison I made was to determine if the link embedded in the email was actually pointing to the Web site it said it was, and not apparently a link with a different URL within it.  In this particular case, the link was to a Google reader URL, and did have some objectionable content. So, although I can’t very well block any messages that might have Google reader links in them, you might be able to.  It depends on your email use policy and Internet access policy. Perhaps your business and your employees just have no use for Google reader at work. If not, I found several more spam messages that got through, with completely different text content, completely random and almost literary, with no obvious mention of sexual content, all sent from major web based email services.

The common relationship was the inclusion of a link that pointed to Google Reader. That’s what we’d need to filter as objectionable content. Other links to other sites came in some other spam emails, but there were enough (three) in a short time that we can see this was the mechanism they were using. The near-random and non-contextual nature of the Google Reader links make just blocking them based on the URL difficult, the ones posted by users have simply long numerical strings as identifiers. Pretty much random as well, although it might be possible rather than blocking any and all links to Google Reader content to selectively block ranges of users, although how to do that efficiently, I can’t yet see.

The Latest Spam Getting Through Your Filtering – and What to Do About It

Free ebook download: Top 10 Most Popular Troubleshooting Posts for Email Administrators