A look inside Facebook’s data center

0/28 Replay Gallery More Galleries
SEE SLIDESHOW

A look inside Facebook’s data center

Facebook invited a small group of reporters to its Prineville, OR data center this week. Most companies don’t even give their own employees access to their data centers (outside of the relatively small group of people it takes to keep the millions of servers inside up and running), but because Facebook open sources most of its server and networking designs, it’s a bit more open to allowing outsiders inside, too. Still, it’s been two years since it last allowed press tour its facilities, so when the company asked us if we wanted to come over for a tour (and lunch), we couldn’t say no.

Prineville was the first data center Facebook built and the complex now consists of three massive buildings (and one smaller one for cold storage). The smallest of the three data center buildings is about 350,000 square feet in size and the newest, which is still under construction, will measure over 450,000 square feet. Each one could easily hold a modern aircraft carrier and still have plenty of room to spare.

Let’s take a look inside one of these buildings.

1/28

No secrets

Other companies don’t put their names on their data centers. But not Facebook. This is the sign at the entrance to the Prineville complex.

2/28

Water and solar

One of Facebook’s innovations — and the reason why it chose the High Desert of Oregon for its first data center — was to use the outside air to cool its servers instead of relying on air conditioning (more about this later). In the background, you can see one of the huge water tanks the company uses to cool the air down when it’s too warm outside. While Facebook draws most of the energy to power its servers from the local grid, it also uses solar to power some of the auxiliary systems.

3/28

Spares

When anything breaks, technicians pick up spares from rooms like this. Every item and every hard drive is barcoded and the chain of custody is maintained at all times.

4/28

Community

In a small town like Prineville, Facebook is a major employer and the company’s presence ensured that others (like Apple) are now also building similar facilities in the town. Facebook also helped upgrade the local electricity and water systems. The local high schoolers sure are happy about that.

Advertisement
5/28

Where your data goes to die

One room we weren’t allowed into — and even most of the data center employees are barred from — is the disc erase room. That’s where Facebook deletes all of the data on its used hard drives before they either get reused or destroyed.

6/28

Snacks

It’s not exactly the Google cafeteria, but you’ll find plenty of snacks on the way to the big server rooms.

7/28

Servers everywhere

Unlike its competitors, Facebook doesn’t rely on air conditioning to regulate the temperature inside, so the inside of the actual server rooms aren’t all that loud and the temperature is pretty comfortable. Over there on the left are the servers that host data about user accounts. If you used Facebook today, there’s a good chance you used one of those machines in the photo.

8/28

More servers

And here on the right are the servers that hold your actual data and manipulate it.

Advertisement
9/28

Down the aisle

Every one of the aisles features over two dozen server racks. These are all Open Compute servers, so you could theoretically build your own.

10/28

A closer look

Here is a closer look at what the servers look like. There is a lot of blue here. Facebook says that’s because blue LEDs were cheaper when it first started building its own servers.

11/28

And another look at the servers

And here are the same servers from another point of view.

12/28

Taking heat

To cool the CPUs, the servers draw the cool ambient air in and then exhaust it into a closed off ‘hot aisle’ behind them. From there the hot air naturally rises and is then exhausted out of the building (more about that later). Unlike the rest of the building, these aisles are hot and loud.

Advertisement
13/28

Hello fans

A closer look at the fans behind the server racks.

14/28

Big Sur

Besides the standard Open Compute servers, the data center also houses Facebook’s Big Sur machine learning servers. Each one features eight high-end Nvidia Tesla cards with M40 GPUs. These are off-the-shelf GPUs and cards, so the servers are a bit bulkier than Facebook’s standard compute machines.

15/28

Inside Big Sur

Here is a look inside a Big Sur servers In the back are the Nvidia cards.

16/28

More Big Sur

And these are the two CPUs that power each Big Sur server

Advertisement
17/28

Cool air

After checking out the server rooms, we went up to the second floor to check out the cooling system. This is what Facebook calls the air intake room. On the right, the outside air comes in and then hits the filters on the left to keep dust out of the building.

18/28

Water

Once the air is filtered, it moves through an evaporative cooling system. When it’s too cold outside, though, Facebook also mixes in some of the hot air from the server rooms to get the temperature of the outside air up.

19/28

Fans

The system can’t run without some moving parts, of course. Here on the right are the fans that suck the air into the server rooms.

20/28

Out with the old

Once the air has done its job, its exhausted from the building using huge fans.

Advertisement
21/28

The building where your 10-year old photos live

Over there is building #4. This building is almost completely dedicated to ‘cold storage.’ When Facebook notices that neither you nor your friends look at your old pictures anymore, it can’t just delete them, after all. So the company then moves the copies of those pictures (or your status updates, videos, etc.) to this building (and others like it around its other data centers around the world). The servers there are essentially asleep until somebody needs data that is stored on them.

22/28

Prineville 3

The third building in Prineville is still under construction.

23/28

Generators

If there is ever a problem with the electricity network, a bank of huge generators sits outside of the building to take over.

24/28

Cold storage servers

Here are the servers that hold all of the old, unused Facebook data. Each one of these racks holds 32 servers and can store 2 petabytes of data (that’s a thousand terabytes — or millions of photos).

Advertisement
25/28

More cold storage servers

Here is a closer look at these server racks.

26/28

Device testing lab

The cold storage building also currently houses Facebook’s mobile device testing lab. These are about 60 self-contained racks that house 32 phones each for testing new versions of Facebook’s apps right on the device.

27/28

Facebook

There is very little signage inside any of the buildings, but you can still spot a stray Facebook logo here and there.

28/28

Bonus: Apple's data center across the street

Apple also has a data center in Prineville. You won’t find an Apple logo anywhere around the complex, though, and I don’t think Apple will ever invite us over for lunch either.

BACK TO
TOP