The team in charge of tracking Facebook’s (FB) growth works on the second floor of Building 17. Most days, the offices are like anywhere else at Facebook: whiteboards, toys on desks, shorts and flip-flops, pretty low-key. Around noon on Sept. 14, the second floor was packed. In one of the common areas, a giant screen showed the number of active Facebook users worldwide. About 100 people, including Mark Zuckerberg and his top lieutenants, watched the numbers run up by about a thousand users per minute: 999,980,000 … 999,990,000 … 1,000,000,000. The counter paused for a moment when it rounded 10 digits, as if to emphasize the point: 1 billion users.
The celebration was less exuberant than one might imagine given that Facebook had just officially registered one-seventh of earth’s population. Zuckerberg had thought about doing the whole balloons-and-visit-from-Ryan-Seacrest thing when they located lucky user No. 1,000,000,000. The problem, though, was that the occasion was really more of a notional event, like when the United Nations announces the world’s population. Facebook’s vast array of computers handles so many users doing so many things, the best they can do is make a statistical calculation. After a few minutes of hoots, high-fives, and good cheer, Zuckerberg and his employees did what they usually do after major achievements: They went back to work.
“I don’t even know if we knew who the billionth person was,” Zuckerberg says about two weeks later. He’s sitting outside at the company’s sprawling Menlo Park (Calif.) campus, resting his arms on a tiny café table. In his usual rapid-fire delivery, Facebook’s 28-year-old chief executive officer explains that his aversion to overt jubilance goes back to the earliest days of his company, when it was still a dorm room operation at Harvard. “We have this ethos where we want to be a culture of builders, right? We don’t want to overly celebrate any particular milestone,” he says. He knew even in college that a company would soon unite a huge portion of humanity via a single social service, he just wasn’t sure it would be his. “We were just these college students, and who were we to build this big thing?” he says. “Clearly, there were other companies used to building software at scale, and one of them would do it.”
Facebook got there first for a lot of reasons, many of them familiar to anyone who saw the movie The Social Network: Zuckerberg’s ambition, knack for addictive widgets, and what some would call supreme ruthlessness. The more impressive reasons, though, have to do with the culture he established, which is expressed in the motivational posters around the company’s offices: “Move Fast and Break Things.”
Facebook absorbed Silicon Valley’s hacker ethos and amplified it. Tech companies normally do controlled beta versions of their technologies; Facebook doesn’t beta anything. It runs as an unending series of quick, on-the-fly tests with actual customers. Engineers race to put up new features, see if they work, and make tweaks to fix them if they don’t. Even trainees who haven’t finished their six-week indoctrination program are asked to work on the live site. The live site, by the way, runs on custom-designed hardware and software housed in Facebook’s superefficient, and experimental, data centers. Every now and again the whole site crashes, but Zuckerberg can live with that. “The faster we learn, the better we’re going to get to the model of where we should be,” he says.
The learn-on-the-go philosophy regularly blows up in Zuckerberg’s face. He and his team periodically revamp Facebook’s privacy policy, triggering a predictable chain reaction: consumer outrage, company walkback, adjusted policy, re-release, lessened outrage, and so forth until the furor dies down. Unlike with computer algorithms that temporarily crash the system, however, these iterations are apt to leave lasting damage to Facebook’s reputation.
Then there’s the initial public offering, perhaps Facebook’s biggest face plant. The company set out to maximize its offering price and succeeded, raising $16 billion when it went public last May. Then it watched the stock sink from $38 a share all the way to less than $20, slicing its market value in half. The foundering stock price has hardened doubts about the strength of its business model, and may make it difficult for Facebook to retain talent. Regulators are looking at whether the banks underwriting the offering selectively shared information with top clients. Multiple plaintiff groups have filed suits in federal and state courts, alleging that Zuckerberg, Facebook, and its investment banks misled investors about the social network’s revenue outlook. Facebook says the suits lack merit.
“The performance has obviously been disappointing. I mean, we care about all the investors, and that’s really important,” Zuckerberg says. “I suppose there could be short-term things that we could do, but we’re not going to focus on those; we’re going to focus on the long-term stuff.”
If the Hacker Way, as Zuckerberg calls Facebook’s culture, is an awkward tool for dealing with Washington and Wall Street, it’s been extremely effective for building a service that can handle the social needs of a billion people. Sites such as Google (GOOG) and Amazon.com (AMZN) have the luxury of preparing the majority of the user experience in advance. Google catalogs the Web and has already decided what to show you before you even search for something. At Amazon, a box of pretzels is a box of pretzels; the pretzel page doesn’t upload videos of itself, crowdsource for beverage pairings, or share wedding pictures.
Facebook builds a custom Web page every time you visit. It pores over all the actions your friends have taken—their photos, their friends, the songs they listen to, the products they like—and determines in two-hundredths of a second which items you might wish to see, and in what order. Each day, Facebook processes 2.7 billion “Likes,” 300 million photo uploads, 2.5 billion status updates and check-ins, and countless other bits of data, and uses that mass of transactions to guesstimate which ads to serve up. To manage the load, Facebook has created the world’s largest, most efficient data-crunching machine.
“I think Facebook has the hardest information technology problem on the planet,” says Mike Stonebraker, a computer scientist and longtime professor at the University of California at Berkeley. “A company like Google certainly does innovative stuff, but Facebook solves the harder problem.” The company is essentially building a system that will serve as the model for the next wave of computing in which trillions of sensors, smartphones, and medical devices issue streams of data that must be analyzed instantly. So far, Facebook actually has this monstrosity working.
Before Facebook moved in last year, the campus in Menlo Park was the headquarters of Sun Microsystems. It pushes up against marshlands and salt ponds along the San Francisco Bay, which gives the air a faint stench when the wind blows a certain way. Zuckerberg nevertheless likes to take meetings outside and today holds forth in the central courtyard. He has a speechwriter now, and his answers have been molded by a few years of intensive media training. “It’s really humbling to get a billion people to do anything,” he says.
Zuckerberg visibly relaxes as the conversation turns to Mike Schroepfer, his vice president of engineering. While much of the world knows Sheryl Sandberg, Facebook’s chief operating officer, they do not know Schroepfer, whose desk is right next to Zuckerberg’s.
Schrep, as everyone at Facebook calls him, arrived four years ago from Mozilla, where he led the development of the Firefox Web browser. No person outside of Zuckerberg has had more influence on Facebook’s engineers and the construction of the company’s massive and growing infrastructure. “He’s the cultural center of the engineering team,” Zuckerberg says. Facebook’s engineers have taken to decorating their pages with Schrep images fashioned to look like the iconic Obama posters from the 2008 campaign. “People just, like, love him,” says Zuckerberg.
It’s Schroepfer who ensures the tweaking never stops. He’s been instrumental in developing the company’s obsession with what’s known in Silicon Valley as A/B testing. This is the process by which engineers compare one version of a Web page to another to see which generates the most interest. You might make one version of a menu red and another blue, then run data analysis to gauge their performance. At any given moment, Facebook has tens of thousands of A/B tests running. It may be trying to decide which version of an ad plays best to a particular crowd or how to word the options in a menu. This is how Facebook discovered that people got stressed out and left the site when they were asked if they wanted to “reject” a friend request. The engineers found the noncommittal “not now” kept people logged on. “Engineers understand data,” Schroepfer says. “The trick is some of these details really matter and some don’t.”
During a meeting in a conference room near his desk, Schroepfer leads a group of engineers in a half-hour debate over the design of a restaurant review feature. Should it have a five-star rating option, a Like button, or both? Should there be animation? Does it feel natural? At the end of the meeting, Schroepfer and one other guy remain at odds over the Like vs. Stars question. “This is probably just a data gap between your brain and my brain,” Schroepfer says. The A/B tests have since begun.
The de facto proctor of these tests is Chuck Rossi. In a company full of people trying to establish their wackiness, Rossi stands out. To the left of his desk he has a large, fully stocked Tiki bar with a lot of Scotch. Used targets have been stapled to the bar—Rossi is a competitive marksman, and sometimes conducts offsites at shooting ranges. “The booze we do separately,” he hastens to add.
Rossi is Facebook’s release engineer, the overlord of a ritual the company calls “the push.” This is the moment when Facebook refreshes its source code with a new version. In the past, it wasn’t uncommon for the push to bring down all of Facebook, but now things tend to run more smoothly. Most software companies take months to pull together the code in their flagship products, check it for errors, and get it ready to ship. Rossi used to have a similar job at Google, and says that famously fast-moving company did its equivalent of the push every couple of weeks while he was there. Facebook redoes its site once a day, usually at around 4:30 p.m. West Coast time.
As push time approaches one evening in August, Rossi is relatively calm. All day, Facebook’s engineers have been sending him new bits of code they want added to the site. The company’s engineers typically submit about 300 tweaks per push. On Tuesdays, there’s a bigger push that incorporates thousands of changes.
Rossi and a handful of other people review the tweaks, approve some, and kick back the rest for improvement. The engineers, proud of their ideas, often try to charm Rossi into approving their code. They gather in chat rooms to try to convince him that it’s crucial that their changes go in today’s push. When tomorrow is absolutely not an option, the engineers leave their desks and make their way to Rossi with an offering—hence the Scotch selection at the Tiki bar. “A little bribe never hurts,” he says. “I love cupcakes, too.”
Facebook has developed dozens of tools to catch the mistakes of its engineers. Rossi’s computer screens—which he’s programmed to look like The Matrix, with green text cascading down a black background—have them all on display. One tool analyzes code to see how dramatic a change is and what mechanisms it will affect. The same program tells Rossi about the level of discussion among product teams about the merits of this new feature. Was it rejected once, fixed, and then accepted? Or did it have 32 comments, 11 updates, and 6 rejections before reaching him? Another application alerts him if something is likely to break. “I actually think as we’ve gotten more mature, we’ve focused a little bit less on the ‘break things’ part of ‘break things and move fast,’” Zuckerberg says. “One of the things that we’ve realized is that as we’ve gotten bigger, it is possible to make so many mistakes that you’re actually moving slower because you’re spending a lot of time fixing mistakes.”
As the afternoon wears on, Rossi decides—Scotch and cupcakes be damned—that he’s got to stop taking new changes. He begins the push by releasing the updated site first to Facebook employees, and then to about 2 percent of Facebook’s users. He brings up another set of windows to see how long the tasks are taking, how well Facebook’s servers are running, and whether there’s any unusual surge of hostile tweets. When something goes wrong, an alarm sounds and the affected window on his screen turns red. “If this board is green, we are good,” Rossi says. A screen flashes green. He patters on his keyboard, and today’s Facebook push goes out to all 1 billion users. “It’s like giving birth,” he says.
From their first day of employment, Facebook engineers are expected to dig into the software that runs the site. Trainees in the six-week Boot Camp sit in a large workroom amid huge banks of desks with massive Dell (DELL) monitors. Under the supervision of “code mentors,” easily spotted by their giant sombreros and other outlandish hats, new hires hunt for minor glitches in the 1.5 gigabytes of source code that runs Facebook.
Some Boot Campers break stuff. Jocelyn Goldfein came to Facebook in mid-2010 from VMware (VMW). Despite an impressive résumé, she still had to go through Boot Camp with the college kids. Near the end of her six weeks, she activated the product she’d been working on, and it flooded Facebook’s internal network with meaningless traffic, taking down the e-mail system for an entire day. “I turned off Mark Zuckerberg’s e-mail,” she says, still shuddering. “This is not how I wanted to introduce myself.” No one reprimanded her. “People showed up and were curious about why this was happening and wanted to know what I was trying to accomplish,” she says. “No one was like, ‘How dare you!’ And that was unique in my experience.”
At some point during their training, the engineers are divided into groups of about 30. They’re taken past a kitchen with free candy bars, a video game arcade, and a vending machine that spits out free keyboards and memory sticks. The recruits end up in a conference room, plop down in plastic chairs, and await a lecture from Schroepfer.
Schrep makes his way to the front of the room and pushes his black glasses up his nose. Of average build and with an orthodontist’s countenance, he’s known as that supersmart guy you want to impress. The recruits—all but a couple are male—go quiet as Schroepfer’s first PowerPoint slide hits the projection screen. He begins by asking the employees where they last went to school or worked: Stanford, Northwestern, Princeton, Duke; CBS Interactive (CBS), Microsoft (MSFT), Yahoo! Research (YHOO), Google, and Amazon.com. “I want to talk about how Facebook is different than any organization you have been in,” he says.
Schroepfer explains that in Facebook’s olden days—that would be 2008—the engineering team had 150 people. Everyone knew each other, hopping from project to project was easy, and the site adapted quickly to competitive threats. The company now has 1,000 engineers and expects to add hundreds more this year. Boot Camp, Schroepfer explains, is an attempt to fight the natural creep of bureaucracy that comes with rapid expansion. He says big tech companies typically spend a day or a week indoctrinating new employees, mostly to explain 401(k) options and how to file expense reports. “Then, boom, you’re deposited in your team.” The employees become part of “warring tribes” that fixate on a single product, he says.
During Boot Camp, the Facebook hires see the company’s entire code base. They’re asked to work on Timeline for a bit and then to try out the Mobile app and familiarize themselves with the inner workings of News Feed. At the end of the process, they choose which product team to join. “We don’t assign you to a team,” Schroepfer says. “We want to make sure you are working on something you are good at and passionate about.” At Facebook, he says, you can come up with a new idea today and “half a billion people will use your product tomorrow.”
Facebook has grown from 600 employees in 2008 to 3,976 as of June 2012. “We have this stat that we throw out all the time,” says Zuckerberg. “There is on the order of 1,000 engineers and now on the order of a billion users, so each engineer is responsible for a million users.”
Even as it expanded its engineering corps, Facebook had to expand its hardware, and three years ago Zuckerberg and Schroepfer decided that the company should stop leasing data center space and build its own facilities. They put together a team of about a dozen people. A few months later, Facebook began constructing a 330,000-square-foot complex in Prineville, Ore., a sparsely populated farming and ranching town that has cheap power and plenty of broadband access.
Building another megadata center wasn’t enough for Zuckerberg and Schroepfer, however. They wanted to experiment with a different design. Instead of using massive air conditioning systems to keep thousands of servers from overheating, the Prineville facility has giant, slatted vents on its sides that let in air as needed. A series of fans, filters, and water misters then clear the air, cool it, adjust its humidity, and guide it deeper into the building. Eventually the air flows to the rows of server racks below. All this happens automatically, with computers deciding how to guide the process.
Most of the software running these data centers was likewise built from the ground up or modified by Facebook engineers. The company has invented ways to get thousands of computers to work in unison. It also has several labs at its headquarters where it designs servers and storage systems. Its engineers strip out nonessential components and test various motherboard configurations to see how much heat and power they can withstand. Instead of buying its computers from a Dell or Hewlett-Packard (HPQ), Facebook has them manufactured by Asian suppliers.
The Prineville data center may be the most energy-efficient in the world. According to a standard measurement developed by an industry consortium, almost all the electricity goes to the computers as opposed to powering cooling systems or dissipating as heat. The company is building a half-dozen similar centers in Oregon, North Carolina, and Sweden.
As those come online, Facebook will be able to handle more Likes, pictures, and status updates, and the ads that go along with them. “There is this Facebook equivalent of Moore’s Law,” Zuckerberg says. “Each year the amount of stuff that each individual shares is growing at this exponential rate. And that lets us project into the future and say, ‘OK, two years from now people are going to be sharing twice as much, [in] three years, four times [as much], four years, eight times as much.’”
The more Facebook grows, the more expansive its services can become. “We are trying to map out the graph of everything in the world and how it relates to each other,” says Mike Vernal, one of the company’s top engineers. The goal, he says, is to record every book, film, and song a person has ever consumed, then build a spectacular model of other things that person could enjoy. Take that vision to its logical end: You show up in a strange city and Facebook tells you what bar to go to. When you get there the bartender has your favorite drink waiting, and you’re able to look around the room and see if anyone there went to your college or likes the San Francisco Giants. You may find this kind of universal social mapping exciting, or creepy. Zuckerberg describes it as inevitable. “At some point,” he says, “that will start to be a better map of how you navigate the Web than the traditional link structure.” By “traditional link structure,” he clearly means the basis of Google searches.
Zuckerberg says he hopes to bring Facebook to China, where it’s currently banned by the government, though he doesn’t think that will happen soon. The company’s biggest growth opportunity is mobile: “There are actually already 600 million people using Facebook on phones, so that’s growing really quickly. And as more phones become smartphones, it’s just this massive opportunity.”
Zuckerberg’s embrace of mobile will strike some shareholders as late in coming. In February, Facebook disclosed in an S1 regulatory filing that mobile usage didn’t generate meaningful revenue. More recently, executives have tried to reverse that sentiment by touting preliminary test results that show the company’s ads on mobile phones are 13 times more likely to be clicked on than ads on the desktop. Renewed faith in the company’s mobile ads has been partly responsible for a mini-rally in its stock, pushing the price above the more dignified threshold of $20 for the first time since mid-August.
“We make more mistakes than other companies do,” Zuckerberg says. “You can’t have everything, so you just have to choose what your values are and where you want to be. For example, Microsoft has a huge focus on really rigorous, bug-free code. That’s cool. I think that’s the right decision for a lot of the markets they’re in. But for us, this is the right way to go.”
In December 2010, Facebook users who were paying attention began noticing something strange about the site. Some people were able to see a new product called Memories, while others noticed a new interface for Photos. The site seemed to be rolling out dozens of products, and many looked unfinished. For about 30 minutes, people filled Twitter with chatter, trying to figure out what was going on.
At its headquarters, people were freaking out. The move-fast culture had resulted in one engineer accidentally releasing all of Facebook’s top-secret projects to the public. Memories, for example, was an early version of what would become Timeline. Most companies keep sensitive projects cordoned off on special systems. Facebook, remember, gives every engineer equal access to its code.
The engineers flew to their internal chat system and tried to devise a plan for fixing the breach. The problem, though, was that Facebook runs on thousands upon thousands of servers. Try as they might, the engineers could not recall the leaked features. Jay Parikh, vice president of infrastructure engineering, decided to take the drastic step of disconnecting Facebook from the Internet. He logged into one of the social network’s central servers and erased all the Internet addresses people would use to get to Facebook.com.
Once he made Facebook invisible to the world, Parikh knew he had to tell Zuckerberg. Not pleasant, even at a company where failure is an option. “I ran upstairs to talk to Mark,” Parikh says. Zuckerberg replied, “That sucks.”
A short while later, Zuckerberg tried to ease the tension. “I got all my direct reports together,” he says. “They were all a little nervous. I was like, ‘Guys, I told you NOT to ship everything. Maybe I’ll say the “not” a little louder next time.’”
