Troubleshooting SMTP/TLS with OpenSSL
Written by Ed Fisher on September 7, 2010
Encrypting transmissions between servers using SMTP/TLS can protect email from prying eyes, but as with all transport layer encryption, it makes troubleshooting issues on the network much more difficult. With WireShark, or even running debug at the firewall, it is relatively easy to diagnose issues by simply observing the SMTP messages between mail servers to see if there are problems. When those messages are encrypted, your visibility into the network goes away, and you are left with event logs and other diagnostics. One of the more common problem areas you run into when using SMTP/TLS has to do with the certificates in use. Unfortunately, this also results in some of the most cryptic messages you’ve ever seen in event viewer, syslog, or /var/log/mail.d.
Fortunately, you can do some fairly low level troubleshooting of certificate issues with a little OpenSSL magic. If this sounds useful to use, please, read on.
Since most enterprise email systems support SMTP/TLS as a method to securing email transmissions using PKI certificates and the secure exchange of keys, the likelihood of you having to support this is on the rise. As with any other encryption, the tools most admins with a networking background turn to, like protocol analysers, become less and less useful because the very encryption that is intended to secure email also occludes the data exchanges between servers.
Since SMTP certificates are the root of SMTP/TLS, and have to be ‘just so’ before two servers can move on to exchanging email, we need to ensure that they are good to go before moving on to other checks.
To successfully use SMTP/TLS, the certificates used must be
- appropriate for the protocol,
- support digital signatures and key encipherment,
- the CN or SAN must match the hostname of the server presenting the certificate,
- and the client must (usually) trust the issuing CA.
If you are obtaining certificates from a public CA this is usually straightforward, but since more and more organisations are rolling their own to save money, this is not always so straight-forward.
While I roll my own at home, and often will do so at the office for strictly internal purposes, I always recommend that for an enterprise, use only certificates issued by a commercial CA whenever you are supporting connections from clients, customers, or other third parties. The few dollars spent on a certificate from a trusted CA is far cheaper than the efforts troubleshooting issues, or the negative impression a potential customer may get when presented with a warning about your certificate!
You can view the Client Hello and certificate download in WireShark, but sometimes it is easier to just drop to the command line. While Linux based MTAs will have OpenSSL installed already, Windows admins will want to go download OpenSSL for Win32 from Shining Light Productions. First, install the Visual C++ redistributables (linked from that site to MSFT,) then install OpenSSL. Once you have it installed, the commands are essentially the same on Windows and Linux, but we’ll show the Windows cmd prompt syntax below.
- open a cmd prompt or terminal session.
- change to the OpenSSL directory. In Windows that will be C:OpenSSL-Win32bin by default.
- issue the following command, substituting the appropriate hostname, and you want to test port 587 as well.
c:OpenSSL-Win32bin>openssl s_client -connect demeter.retrohack.com:25 -starttl s smtp Loading 'screen' into random state - done CONNECTED(00000180) depth=0 CN = apollo verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = apollo verify error:num=27:certificate not trusted verify return:1 depth=0 CN = apollo verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/CN=apollo i:/DC=home/DC=olympus/CN=olympus-CA --- Server certificate -----BEGIN CERTIFICATE----- MIIEzzCCA7egAwIBAgIKHJPqzQAAAAAAGDANBgkqhkiG9w0BAQUFADBEMRQwEgYK CZImiZPyLGQBGRYEaG9tZTEXMBUGCgmSJomT8ixkARkWB29seW1wdXMxEzARBgNV BAMTCm9seW1wdXMtQ0EwHhcNMTAwNDMwMTc1MTQ3WhcNMTIwNDI5MTc1MTQ3WjAR MQ8wDQYDVQQDEwZhcG9sbG8wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALZ8 UOsppbvv7hZa6nxaFZAiQ1MKvY+bAUq3MZ7x6KhDIF/rl+9giuEHn+eX4FM81O1L fZMK6By/FWaBxMK7Bd51R66csrIUr04JFwAYeCEAx+MYn2WImHPAVF5d5o8dHg09 IbG3vXquNDKEKIloR/9gnhogfi4szIxI/rDBdp69AgMBAAGjggJ4MIICdDAhBgkr BgEEAYI3FAIEFB4SAFcAZQBiAFMAZQByAHYAZQByMAsGA1UdDwQEAwIFoDATBgNV HSUEDDAKBggrBgEFBQcDATAvBgkrBgEEAYI3FQoEIjAgMAoGCCsGAQUFBwMBMAoG CCsGAQUFBwMCMAYGBFUdJQAwHQYDVR0OBBYEFNiXwZgWEG63djUQAJa7s0oCRVvh MDMGA1UdEQQsMCqCFyouZWRhbmRjb25uaWVmaXNoZXIuY29tgg8qLnJldHJvaGFj ay5jb20wHwYDVR0jBBgwFoAU8SoB0nFVGBwvTpNt7D5SYQIobnIwgcYGA1UdHwSB vjCBuzCBuKCBtaCBsoaBr2xkYXA6Ly8vQ049b2x5bXB1cy1DQSxDTj16ZXVzLENO PUNEUCxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1D b25maWd1cmF0aW9uLERDPW9seW1wdXMsREM9aG9tZT9jZXJ0aWZpY2F0ZVJldm9j YXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9pbnQw gb0GCCsGAQUFBwEBBIGwMIGtMIGqBggrBgEFBQcwAoaBnWxkYXA6Ly8vQ049b2x5 bXB1cy1DQSxDTj1BSUEsQ049UHVibGljJTIwS2V5JTIwU2VydmljZXMsQ049U2Vy dmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1vbHltcHVzLERDPWhvbWU/Y0FDZXJ0 aWZpY2F0ZT9iYXNlP29iamVjdENsYXNzPWNlcnRpZmljYXRpb25BdXRob3JpdHkw DQYJKoZIhvcNAQEFBQADggEBADB1neeO9cReyIxyEtEQpyn/nEeeRZ6G4x6aqflB /m0YnpIN5C22vQ2FuANaHsJNVi/9U0B5b20V18lM5+6AjMBizadGUv3jcH+jsfT/ JsHiY0C9NEE6kCUlQfD4YuPjiQvnyGjVVVK3UrIaM4YhH4qXZs21qsCWbaGkybIM uk+7viMm1dJoXOHW88ihdqYOwMNxBMbqd61BSWUfn580QV+T9uvz/Q1PF8e8k6Hp B8VWbirUh25CfkLdwe2M2Ys1Z+6AppsDf/Y1DQgHgnacWLv784IQBQjHQ45aEHvl fwkF3YlWKf7X2iAYmJI0SpwKErFKp8OuziisJi3qKI2NCCU= -----END CERTIFICATE----- subject=/CN=apollo issuer=/DC=home/DC=olympus/CN=olympus-CA --- No client certificate CA names sent --- SSL handshake has read 1783 bytes and written 443 bytes --- New, TLSv1/SSLv3, Cipher is AES128-SHA Server public key is 1024 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : AES128-SHA Session-ID: 44030000FDADD13038C7CECB27978C32091B850DAC5FF0B39E6BD48CFE3B7560 Session-ID-ctx: Master-Key: D505277A00EF304B3DF7FE99E72618532A616ABE5A40FA26D0B73647E44BB4BB 08515582DDD288306B08A51221D21D61 Key-Arg : None PSK identity: None PSK identity hint: None Start Time: 1283360267 Timeout : 300 (sec) Verify return code: 21 (unable to verify the first certificate) --- 250 XSHADOW 451 4.7.0 Timeout waiting for client input read:errno=0
The above example shows an exchange with a server where the admin (me) rolled his own certificate. From our client, we can see that we do not trust* the certificate issuing CA, /DC=home/DC=olympus/CN=olympus-CA.
*OpenSSL on Windows does not use the Windows certificate store, so even if you are running this on a domain joined machine and issued the certificate from an AD integrated PKI, you will still get this error unless you add the CA to OpenSSL’s list of trusted CAs.
Unless our server is configured to accept certificates from untrusted CAs, we are going to either have to trust this root CA, or have the admin of that mail server obtain a certificate from a trusted CA.
See that we can also see the entire certificate. That can help with further troubleshooting name mismatches, etc. Copy everything between —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– to a separate text file, save it as a *.cer, and you can open it to view the certificate. From there we can see that this certificate has two wildcard values in the SAN, which should match to the FQDN we connected to.
If we attempt to connect to a server:port that does not support SMTP/TLS, we get this.
C:OpenSSL-Win32bin>openssl s_client -connect mail.global.frontbridge.com:25 -s tarttls smtp -status Loading 'screen' into random state - done CONNECTED(00000180) didn't found starttls in server response, try anyway... 8972:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:.ssls23_lib .c:184: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 299 bytes and written 254 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE ---
Connecting to a server whose CA we trust, and whose certificate has no problems generates a lot less text.
C:OpenSSL-Win32bin>openssl s_client -connect mail.global.frontbridge.com:587 - starttls smtp -status Loading 'screen' into random state - done connect: No error connect:errno=0
With the above in mind, you should be able to easily identify or eliminate certificate issues when troubleshooting SMTP/TLS.
Posted in email management, email security | No Comments »
