What it's like inside a Facebook data center
Earlier this week, the world's largest social network didn't work. So how *does* it work exactly? I got a peek inside Facebook's North Carolina nerve center years ago, and found out what's changed.
Hey, remember when Facebook went down for like six hours on Monday?
It was somethin’, alright. Basically there was an issue that, in so many words, kept the wider internet from finding Facebook’s servers. It was, in one telling, as if Facebook had been erased from the internet’s phone book. Here’s me explaining what a phone book is to the youths:
There’s been quite a bit of focus lately on Facebook’s, um, corrosive effects on American society. But Monday’s incident brought a little more focus on how the internet itself, as well as Facebook, physically functions. Here’s a bit from a blog post from Facebook’s engineers, explaining the outage:
When you open one of our apps and load up your feed or messages, the app’s request for data travels from your device to the nearest facility, which then communicates directly over our backbone network to a larger data center. That’s where the information needed by your app gets retrieved and processed, and sent back over the network to your phone.
The outage meant that engineers couldn’t fix the problem remotely, so they had to actually travel to the data centers in person to do the thing that we always do when something goes wrong with our computer: turn it off and turn it back on.
But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online.
It is, you see, very hard to get inside of one of Facebook’s data centers. Which is funny, at least to me, because I’ve been inside of one in North Carolina. Twice.
You’re a big data center, yes you are, yes you are
I know! Who’d let someone like me into their big computing hub? Well, you have to remember that it happened in 2012, when Facebook was seen mostly as a wunderkind and largely hadn’t been portrayed as a looming menace to society. I was working for WCNC-TV in Charlotte at the time, and I emailed somebody at Facebook to ask if I could have a tour. After a bit, they said, yeah, sure. Come on by.
So I drove out to Forest City, North Carolina, about 90 miles west of Charlotte, for a look around. First off, the place is enormous. The original data center there is a quarter-mile long, and covers 350,000 square feet. Air is whirring around to keep endless racks of servers cool. It sounds like you’ve stepped into the world’s largest white noise machine. The lights on the server racks blink Facebook blue. There was, at one end, a set of machines that was locked into a black metal cage. That is where all of the credit card transactions took place, I was told.
The scale of the place that was mind blowing. Each server rack could hold billions of pictures. And those racks went on as far as the eye could see. At the time, every picture ever uploaded to Facebook was inside. There were 100,000 of them. But the actual servers themselves were… boring. At one point, my friendly guide tried to liven things up by pulling out a tray of hard drives and disconnecting them. No, he joked, he didn’t just take someone offline, because every picture and status update were backed up in several different places.
Ironically, my original story was … wiped from the internet when my old station redesigned its website. But no worries! Because in 2014, I went back and got another tour, and this time I put it on YouTube:
By that time, Facebook had already built another building right behind the original to house all of the data that was piling up, and then built a third building behind that one. It was called “cold storage,” and it existed basically to hold all of the old pictures that people uploaded but never really looked at anymore. At the time, engineers there said 82 percent of Facebook’s traffic went to only 8 percent of its pictures, which means there are trillions of images there. Nobody views them. But nobody deletes them, either.
I used to feel like that building was proof that on the internet, we’re all hoarders. But hoarders, at least the TV reality show version of hoarders, pile up all of their stuff around them and fill their own houses. It was only a few years later, when I took some bulky items to the Guilford County dump, that I felt the same feeling that I did when I toured cold storage. Oh God, I thought, with a feeling of dread, seeing huge piles of trash. This is where it all goes.
Power, Unlimited Power
Today, stories about Facebook are really stories about power. But back then, stories about Facebook’s data centers were about … power. Specifically, electricity.
In 2012, Facebook’s single data center in North Carolina used enough electricity to power something like a thousand homes. As a backup, 14 tractor trailer-sized diesel generators were there to keep the whole place running if the power ever went out. And, again, this was just for the original data center. Facebook now has a total of four giant buildings in Forest City, all full of whirring, humming servers that run all day, every day.
This may seem obvious, but you know who loves data centers in spite of the fact that they generate only a handful of jobs and—other than some charitable donations—don’t really make a huge impact on the local economy? That’s right! Power companies! I wrote about this in 2011, when Apple was opening its data center in Maiden, near Hickory.
Duke Energy, for one, welcomes our new Apple overlords. Executives gushed about it in a release shortly after the company announced it was coming to Catawba County. “The great thing about a data center is that they run full‐out, 24‐7, with no shifts and no seasonality," said Clark Gillespy, vice president of Economic Development, Business Development and Territorial Strategies for Duke Energy Carolinas. "It's the type of customer where the meter spins and spins at an exponential pace. It may be the most ideal customer we could have."
"We fully expect Apple to be one of our top ten customers in the Carolinas," said Stu Heishman, Duke's director of Business Development. Duke also lobbied to get the Google and Facebook server farms, and supplies power to them both.
Shortly after that story came out, Duke pulled its press release off of the internet.
It’s unclear exactly how much money the power company makes from data centers, but up in the foothills, Duke Energy had a heavy power grid that once powered electricity-sucking furniture factories. Those factories are largely gone, which meant Duke had excess capacity in those areas. The high-tension wires were already in place. In Apple’s case, those wires run right behind its data center. You can see in images that the one thing that sits between every power line and every data center is a very large substation. In Apple’s case, there are several.
Another reason? Lenoir, Maiden, and Forest City are all in the Isothermal Belt, an area of milder temperatures. That’s important for data centers, which generate a lot of heat. But overall, the big reason is the cheap power here in North Carolina. Add in some state and local tax incentives and you’ve got yourselves a bonafide data center cluster out there in them thar hills.
You know who wasn’t happy to see all of these burgeoning tech companies setting up shop in North Carolina? Greenpeace, that’s who. In a report in April 2011, they pointed out that our state’s cheap power was cheap because it came largely from coal and nuclear power, which was dirty and potentially dangerous, respectively.
What About Now?
Luckily, we’ve reined in our reliance on the internet and kept everything at a sustainab— JUST KIDDING EVERYBODY BUILT A SHIT TON MORE DATA CENTERS.
Google is still there in Lenoir, and there’s a decidedly Google-y touch: On Google Maps, you can take a virtual Street View tour around the outside and inside of the data center, even though the road around it is behind a gate and off limits. Fun fact: It’s NASCAR themed!
Disney has a large data center in Kings Mountain, one that’s been up and running for about ten years and was expanded to help run its Disney+ streaming service. Boeing has a data center in Kings Mountain too.
Apple, for one, has added a ton of solar power capacity right across the road from its data center, which was a part of then-CEO Steve Jobs’ promise to make the center, and the iCloud that it powers, “as eco-friendly as you can make a modern data center.” The only other thing out there is a quintessentially North Carolina thing: A Hardee’s behind a gas station.
And then there’s Facebook. Forest City, which opened in 2012, was Facebook’s second data center. Today, there are a total of 18 worldwide, with a 19th under construction. Each, per Facebook, is supported by 100% renewable energy, which means they’ve added enough stuff like solar power to the local grid to offset their power usage. That’s significant, but keep in mind that North Carolina has long required power companies to hit energy conservation goals (this is why, for a while, Duke Energy would send you a bunch of energy-efficient light bulbs for free). It’s also required more and more power from renewable sources. Those regulations are about to get even stricter, it seems, but things have been trending away from coal and toward renewables for a while now, both nationally and in North Carolina:
North Carolina’s coal usage is less than half of what it was in 2012. So yes, data centers are using an increasing amount of renewable energy. We all are.
Still, prices in North Carolina remain lower than the national average. And industrial rates are, and have always been, lower than the rates that you and I pay to keep the lights on at home.
But enough about power. More than anything, a trip to a data center is a reminder that the virtual world isn’t quite as virtual as it seems. The cloud is, actually, a real place, one that gobbles up electricity to store stuff that seems limitless. Facebook can’t build billion-dollar data centers fast enough. And on Monday, we were reminded that our connection to it is tenuous. It can go away. Quickly.
Back in 2012, I asked my Facebook tour guide if he could show me where the internet came into the building. I was expecting something the size of a New York water main, but instead, he took me to a spot in the corner of the center where a set of cables as wide as my arm came up out of the concrete floor. That was it? I thought it would be bigger. Protected by a metal covering or something. It struck me that I could take the entire data center offline with a big axe or a chainsaw.
What happens if I cut that, I asked.
It would be very bad, the guide said.
Would it, though?
Definitely food for thought. I liked your take on things and the humor you threw in here and there.
Pulled The Plug On FB In 2015.
My Brother And I Were Living In The Same House Together.
Somehow FB Combined BOTH Of Our Accounts.
I No Longer Received MY Updates And Only Received HIS Updates.
Never Enjoyed Those Pesky Algorithms Directing Ad Plug - Ins To My Page.
As The FB Business Model Expands To Generate Ad Income, Global Societies Will Continue To Be Monitored 24/7.