There are certain ways of doing things in hardware engineering, and engineers simply follow these rules because there’s no use fighting them even if they wanted to. Frankly, most don’t even think about it because it’s just a given, but during a recent tour of Facebook’s hardware lab, director of engineering Matt Corddry, says Facebook scale requires them to rethink the old rules and let engineers imagine outside industry standards.
Since Facebook has begun building much of its own hardware, that means engineers can rethink how things are done, and when you build equipment for the scale of Facebook, it requires creative thinking. “We understand our challenges, costs, operating environment and needs better than an outside vendor and we are able to specialize on the specific needs of Facebook,” Corddry told me recently.
Corddry explained that one of the ways Facebook encourages this creative thinking is to get engineers to work across specialties and talk to one another. “What I find is you need to give engineers learning experiences to find optimizations and break boundaries to get discussions across disciplines,” he said.
Facebook has found when engineers work together instead of in isolation interesting things begin to emerge. “Many silo these engineering teams –server, storage, database, [and so forth]. We don’t create these barriers,” he said. “We bring different teams together to find new ways to solve problems.”
Another thing they do, that many companies, even hardware manufacturers fail to do is they bring their engineers into their datacenters and they watch how people maintain the equipment they design. When you see someone taking out 6 screws (or 16) to replace a hard drive, and you need to do this hundreds of times a year, you begin to get that there needs to be a simpler way.
That’s how they designed the disk array shown here. It’s designed for easy maintenance with no screws. You simply flip the large green lever to slide out the array, pop the small green lever in front of the hard drive you need to replace, lift the hinged lid (as shown below), tug out the hard drive and pop it in a new one. There are no screws involved at all, and when you have Facebook scale, you need to be thinking about these types of issues.
He said that the disk array design I looked at had been iterated over time to make it as simple as possible to maintain. It’s possible they aren’t done yet.
In contrast, Corddry told me he spoke to one hardware engineer at unnamed vendor who admitted he had never watched a technician try to repair his design. When you don’t think about maintenance, it shows and we’ve all dealt with equipment with way too many small screws and poor placement. You end up with bloody knuckles from scraping against the inside of the machine and it’s not fun.
When you’re dealing with a few machines, that’s troubling. When you’re dealing with thousands or even tens of thousands, it takes on a different dimension.
Another way that the Facebook engineers thought outside the box (literally) was the server design. Corddry told me engineers are conditioned that the server design has to fit into a 2u rack, but he gave them permission to forget about that and imagine how they would design a server if there were no rack limitations.
When they left those limitations behind, it opened up all kinds of possibilities. Left to their own design devices, the engineers came up with a long narrow box. Facebook designed a rack to accommodate the new size. You can afford to do that when you’re Facebook. The box slides out of the rack and the top slides off –again no screws –and as you can see from the pictures below, the engineers laid out the box so that you could see the different parts at a glance. As one hardware engineer friend who looked at these pictures pointed out, some of the design principles still hold in terms of where you sit memory in relation to the CPU and the distance you want information to carry, but once again Facebook allowed its engineers to break the design mold and do what felt right.
Corddry said the idea is to put specialists with a narrow and deep focus together with broad generalists in a hackathon-style approach and let them have at these design problems and they really do come up with creative ways to solve what would have been much more difficult design problems.
It’s worth noting that when they are done producing these unique forms of hardware they open source them to the Open Compute project where the designs are delivered to a community of designers to further attack the scale computing problem and figure out ways to produce hardware that’s easier to maintain, runs much more efficiently and can be managed more cost-effectively throughout its lifecycle.
Comment