How Hotmail changed Microsoft (and email) forever

Twenty years ago this week, on December 29, 1997, Bill Gates bought Microsoft a $450 million late Christmas present: a Sunnyvale-based outfit called Hotmail. With the buy—the largest all-cash Internet startup purchase of its day—Microsoft plunged into the nascent world of Web-based email.

Originally launched in 1996 by Jack Smith and Sabeer Bhatia as “HoTMaiL” (referencing HTML, the language of the World Wide Web), Hotmail was initially folded into Microsoft’s MSN online service. Mistakes were made. Many dollars were spent. Branding was changed. Spam became legion. Many, many horrendous email signatures were spawned.

But over the years that followed, Hotmail would set the course for all the Web-based email offerings that followed, launching the era of mass-consumer free email services. Along the way, Hotmail drove changes in Windows itself (particularly in what would become Windows Server) that would lay the groundwork for the operating system to make its push into the data center. And the email service would be Microsoft’s first step toward what is now the Azure cloud.

Former Microsoft executive Marco DeMello, now CEO of mobile security firm PSafe Technology, was handed the job of managing the integration of Hotmail as the lead program manager for MSN—Microsoft’s own answer to America Online. In an interview with Ars, DeMello—who would go on to be director of Windows security and product manager for Exchange before leaving Microsoft in 2006—recounted how, right after he was hired in October of 1996 to manage MSN, he was summoned to Redmond for a meeting with Bill Gates. “He gave me and my team the mission of basically finding or creating a system for free Web-based email for the whole world that Microsoft would offer,” DeMello said.

You’ve got mail

In 1996, the Web was still gaining traction. Almost all personal Internet access was over dial-up services such as AOL, MSN, CompuServe, and EarthLink. A lucky few had early “high-speed” Internet service over ISDN connections, but many companies hadn’t even connected their corporate email systems to the Internet yet. While there were a few Web-based mail offerings from ISPs integrated into Web hosting accounts, and Lotus had demonstrated a Web interface to cc:Mail in 1994, Hotmail and competitor Rocketmail (which would later become Yahoo Mail) were the first to offer free, Web-based email funded by advertising. By 1997, Hotmail already had 9 million users.

“I made the point, and it was obvious,” said DeMello, “that we could not build our own Web mail service in the time that Bill [Gates] had specified.” Buying an existing service was the only real choice—albeit an unpopular one among other Microsoft executives, who usually adhered to the policy of “eating our own dog food.”

But in the end, “Bill wrote a check for $450 million in cash,” DeMello recounted. “And I was given the responsibility of integrating that system and scaling it within Microsoft.”

Vendor lock-in

That responsibility would include the somewhat delicate task of incorporating software running on Unix—a mix of FreeBSD Web servers on the front end and Sun Solaris on SPARC on the back-end—into a Windows-only environment and migrating the service to Windows servers.

Windows NT Server was not up for that task in 1997. While DeMello’s team developed some interfaces to the Windows environment for the Hotmail platform, “we were a customer of Windows Server,” he said, “and at the beginning we were a not very happy customer.”

Despite pressure to immediately move the code to Windows, DeMello said, “There were a lot of things that we were poking at—from security to memory management, and all the way to the TCP networking stack itself—that we were comparing—’this is what we get from Unix, this is what we’re getting from NT and this is why we can’t migrate yet.’ It was always, ‘Nope, we can’t migrate yet.'”

At a time when Sun CEO Scott McNealy regularly made Microsoft’s server operating system the butt of jokes, this was likely salt in the wounds of Microsoft executives. To change that “nope” to a “yes” would take three years and the development of Windows 2000 Server. DeMello’s team “worked with [Windows NT architect Dave] Cutler and crew at the time,” DeMello recounted, “first on the scalability piece—we’re talking about Internet Information Server, and the networking stack, and the TCP stack and memory and how it was managed—and also the security of accessing local folders straight from the executable process. Eventually Cutler and his team were able to pull it off.”

That relationship between Microsoft’s server-development team and the Hotmail team would continue for years, especially for development of IIS, Windows’ Web and Internet services component. “We would have builds that were created to test IIS—Hotmail was always a test bed,” DeMello said. “The mantra was if it passes the Hotmail test, you can give it to anyone—it became a stress test for IIS.”

The operation of Hotmail gave Microsoft the ultimate “eat your own dog food” experience when it came to day-to-day operations of a global Web-based service—experience DeMello believes is reflected in how Microsoft runs the Azure Cloud today. “It was a sort of a bottomless wealth of information in terms of what to do and not to do—best practices, worst practices, what works and what doesn’t,” he said, “from the minute issues of response time on a login all the way to how you’d handle large data transfers.”

While the migration to Windows Web servers happened earlier, the backend system of Hotmail—the database servers and storage—didn’t even begin to move to Windows Server and SQL Server until 2004. The migration became an increasingly heavy lift as storage demands increased, because there were limits to how quickly accounts could be moved from one database to another and be propagated across data centers.

Hotmail also left a mark on the Office platform—aside from being the predecessor to Outlook.com. The first release of Outlook came just a few weeks after the Hotmail acquisition, and the next version—Outlook ’98—had to be adapted to work with Hotmail—leading to a bit of a war of protocols. “[Outlook] was using MAPI [the default interface for Exchange] as a protocol,” DeMello said, and he described MAPI over TCP/IP as “one of the heaviest things ever invented, so we had to change that to straight WebDAV back then. So we had a few issues, let’s put it that way—which protocol had to win the protocol wars.”

The pain of experience

The migration from Solaris to Windows took three years to complete. And while that migration went off largely without incident—DeMello said a “commandment from Bill Gates from above” was “‘Thou shalt not lose a single mailbox’—and we didn’t.” There was still some pain along the way.

Scaling up to serve millions of users meant scaling up datacenters that could handle the ever-mounting storage and compute demands of Hotmail. Storage was far from cheap. “We were dealing with effectively skyrocketing costs for hard drives,” said DeMello. “You have to remember we’re talking about 1997 into 2000… you were still paying through the nose per megabyte—forget about gigabytes. And so the infrastructure cost itself was a staggering bill.”

And those data centers were expensive and power-hungry. “I recall when we actually had finished the new data center, which was built in Bothell [Washington],” said DeMello. “We powered it up to test it—and the first day we tested Saturn, we caused a blackout in Bothell. I had to respond to a very angry city official the next morning. We did pull it off the second time—there was no blackout. The capacity had been upped, and everyone was ready for it and braced for it and expected the city to be licked with flames, but it didn’t happen.”

Then, in the summer of 1999, Hotmail had its first big security breach. Every single one of Hotmail’s accounts—which at the time numbered around 50 million—was potentially exposed by a bug in a script on Hotmail’s servers that gave access to any Hotmail account with the same password: “eh.”

Gateway websites sprang up that used the exploit to allow anyone to gain access to a mailbox by just entering the targeted account name. Some claimed to have access to accounts via the bug for nearly two months before Microsoft patched it. Some believed it was a backdoor left by a Hotmail developer.

DeMello would not comment on that breach. “I could tell you, but I would have to kill you,” he joked. But he contended that Hotmail had always put security and privacy first—at least, as much as was practical at the turn of the millennium. “We put a lot of energy and effort into security and privacy,” he said. “It wasn’t an afterthought. I think we built the system from the ground up focusing on security and privacy.”

For 1999, that meant doing two things especially, DeMello said. “We tried to protect credentials and enforced password policies. And we wanted to be very forthcoming to users about the need to protect their passwords and made it clear that email is not a secure medium. On FAQs, and in communications from the Hotmail team itself, we warned never to share or send any personal or financial information or security info over email.”

Hotmail used Secure HTTP (HTTPS) with SSL encryption to protect users’ login credentials, and Microsoft forced customers to use more complex passwords—but the rest of the service ran over unencrypted HTTP. “Just the authentication piece required us to run hardware accelerators at the time,” DeMello said. “And that had a very high cost—thousands of dollars per card, which you had to run whether you used Unix or Windows Server. You could not run the entire infrastructure at the time over SSL.”

That changed as the CPUs running servers evolved—and today, it’s “unfathomable to run something with straight HTTP,” DeMello said.

Password policies were set up to prevent customers from using passwords that were too short or (starting in 2011) too commonly used. However, Hotmail had a password length limit of 16 characters, so there was a ceiling on just how complex those passwords could get.

So while someone listening to the coffee shop Wi-Fi network might not necessarily be able to sniff passwords, there was still the possibility that someone could read your Hotmail messages by grabbing Web traffic after logging in.

The heartbreak of Hotmail stigma

Competition from Google’s Gmail and from Yahoo forced Hotmail to get better, but it also triggered some weird rebranding. As part of Microsoft’s attempt to make MSN more “live” around the time of the Windows Vista launch in 2005, Microsoft attempted to rebrand many of its services as “Windows Live.” Hotmail was renamed “Windows Live Mail.” But Hotmail users were apparently confused, so they changed it again—to Windows Live Hotmail. Along with the rebranding, Microsoft began a full rewrite of the front-end systems for Hotmail, which had previously been mostly ports of the original Solaris code in C++ and Perl. The rewrite, in C# and ASP.NET code, finally brought an end to Hotmail’s Unix legacy and, for better or worse, made the service a showcase for Microsoft’s own platforms—setting the company on a course toward the Office 365 platform and the Azure cloud.

While Hotmail was important to Microsoft as a testing ground for many things—and perhaps less important as a revenue generator—it also attained a reputation in some quarters of being the root of all that was bad on the Internet. Hotmail users were the butt of jokes and general hate for years. One management consultant openly suggested that companies should never hire people who use Hotmail.

Hotmail was the land of burner accounts for people setting up fake dating profiles. As a pioneer in HTML email, Hotmail users were a natural target for emerging phishing and drive-by download attacks. Its spam filtering capabilities were questionable at best. Ironically, Hotmail’s inability to block spam made Hotmail accounts more likely to be blocked as spam—in part because of all the bouncebacks caused by full mailboxes.

So, despite all the relatively good things we can credit Hotmail with helping along, there’s not a lot of reason to mourn its passing. Outlook.com makes forgetting the bad old days of webmail easier… and there are still thousands of people who were too lazy to opt out of keeping their Hotmail.com address.

https://arstechnica.com/?p=1237983