John C. Dvorak rails against the LAPD’s $7 million dollar email deal with Google. I agree that $7 million is a lot of money to spend on email, but I don’t know all the details of the deal, so I’m not going to cast too many aspersions there. I will, however, take Dvorak to task for his knee-jerk reaction to “hire three programmers and an IT guy”. End users expect email to just work, and have been blissfully shielded from the technical complexities that make running a robust email platform quite challenging.
Email is not an instantaneous protocol
The single most common question I get as an administrator is “Why did this email take so long to get there?” Users are so blessed with lightning fast email delivery when everything is working correctly that they’ve never really had to understand that the email RFCs specify delivery retry times in terms of days, not just hours or minutes. Your email server can work diligently to re-deliver a message for up to three days, without you having to do anything about it. This is a real benefit to system administrators who need to take their email systems offline temporarily for upgrades or maintenance: it means that mail will queue up in all the systems around the world that are trying to communicate with your system, and all that queued mail will flow into your server once you get it back online.
Email servers are complex
We’ve come an awfully long way from the days when your choices for running an email server were between Sendmail, Postfix, and Exim. Things like Scalix and Zimbra make installing and managing a robust email infrastructure easier than ever before, but it still takes skill and savvy to manage upgrades and keep systems online. Incoming and outgoing SMTP queues, vacation messages, content scanning, aliases and redirection, mailing lists: all of these things come to bear on the simple email message that goes from sender to recipient. It’s like the internal combustion engine of your car: you don’t see the chemical reaction that creates the small explosion that moves the pistons; instead you push a pedal and the car moves forward like magic.
Redundancy is not cheap
You might get three programmers and an IT guy to prop up an email service for you. How reliable do you think that email service will be? Email is important — you expect instantaneous delivery, right? — so you need to build in some fault tolerance. You’ll need much more than just a pair of mirrored hard drives to store your mail spools. For any real volume of mail accounts, you’ll likely want to get a Storage Area Network with redundant fibre connections to your servers. That’s right: servers, plural. You’ll almost certainly need more than one. In your SAN you’ll want many hard drives: the more spindles across which read and write operations can occur, the better your performance will be, and the safer your data will be. You’ll likely be using RAID 6 to protect data from multiple simultaneous drive failures. You’ll need battery backups, support contracts on all your hardware, and probably at least one backup IT guy to account for the proverbial “hit by the bus” scenario. Also, your IT guy will likely want to take a vacation, or may get ill occasionally.
Security is important
The skills necessary to build and maintain this email system are not necessarily the same skills necessary to keep this service safe from attack. While all sysadmins should have some modicum of understanding of network security, it takes a special kind of mind to really account for security in depth. You can cut corners and get a firewall appliance with some deep packet inspection, but you’re adding another level of complexity to your setup, with more stuff getting in the way of your email delivery. Someone needs to read log files and respond to incidents. Someone needs to evaluate and apply security fixes (which may result in down time). Your mail server will be the target of regular attacks. Many of these will be childish and trivial and fail immediately. Some will be sophisticated and hard to detect. You don’t want your mail servers getting blacklisted as the result of a compromise, do you? Who’s going to get you un-listed from those blacklists?
Backup and recovery
You’ll need to take backups of your email system, to ensure you can recover from disaster. There’s some more expenses for you. And what about legal compliance? E-discovery is kind of a big deal, and many institutions have data archiving requirements to cover their tails in the event of a lawsuit.
Human resources
To quote famed Ohio State football coach Woody Hayes, “You win with people”. Your programmers and IT guys have lives. They’re going to want raises. They’re going to need benefits. They’re going to seek greener pastures over time, and you’ll need to hire competent replacements.
Conclusion: you get what you pay for
John Dvorak might not like GMail, and that’s his choice to make. But to claim that a robust, reliable email system can be cobbled together on a contract basis is pure folly. Google is able to bring an enormous number of resources to bear on the problem of email. They have a robust, redundant network operation that ensures that your email is online and accessible when you want it. They have trained professionals working around the clock to keep servers not just operating, but operating securely.
There are plenty of other organizations offering the same front-facing service — that is, email — but how many of them have the sophistication of Google’s world-wide infrastructure? I’ve no personal experience with Hushmail, which Dvorak mentions, so I’ve no reason to doubt their competency as an email provider; but are they able to operate on the same scale as Google, or Microsoft, or Yahoo?
Maybe $7 million is a lot to spend on hosted email service. Like I said, I haven’t looked at the full details of that deal, so I can’t speak to the value proposition too much. But I can assure you that not taking your email infrastructure seriously, or skimping on your email service to save a few bucks, will only lead to pain in the long run.