What kind of automated testing does Facebook do? – Quora

What kind of automated testing does Facebook do?

How do they make sure they aren’t breaking things in their weekly pushes?

1 Answer

Steven Grimm, Facebook test engineering tech lead

Facebook test engineering tech lead Software Testing

228 votes by Adam D’Angelo, Olaoluwa ‘Ola’ Okelola, Lee Semel, (more)Adam D’Angelo, Olaoluwa ‘Ola’ Okelola, Lee Semel, Robert Cezar Matei, Joshua Levy, David Molnar, Timothy Wee, Randal Truong, Liron Shapira, Charlie Cheever, Brandon Smietana, Matt Jones, Alina Libova, Avram Cheaney, Gustaf Alstromer, Adrian Aoun, Joseph Hsieh, Shanky Surana, Allan Carroll, Breck Yunits, Matyas Tamas, Stephen McMillen, Ben Bleikamp, Caleb Elston, Chris Thomson, Van Nguyen, Alfred Nelson, Ralph Barbagallo, Siqi Chen, Ilya Sukhar, Kartik Ayyar, Anon User, Kevin Li, Aldiantoro Nugroho, Tracy Chou, Chad Little, Ani Ravi, Mark Folashade, Xiang Xiao, Tom Cook, Bryan Liston, Gilbert Leung, Adam Hupp, Alex Kamil, Marc Kwiatkowski, Mikhail Eliseev, Eric Allam, Arjun Banker, Marc Bodnick, Helder Ribeiro, Eric Kerr, Justin Bishop, Sarah Smith, Diego Berdakin, Ashu Mehta, Andrew ‘Boz’ Bosworth, Eric Ji, Girish Patangay, Ernest Mak, Vijay Rayapati, Joel Lewenstein, Michael Chow, Ray C. He, Avlok Kohli, Paul Schottland, Tom Whitnah, Ali Shah, Steven Loi, Chinmay Kulkarni, Luke Shepard, Josh Smith, Jerry Cain, Can Sar, Can Duruk, Chris Pan, Curtis Chambers, Kenny Lau, Brian Collins, Chris Ballance, Kyan Pardiwalla, Yoann Padioleau, Erik Johnson, Spencer Ahrens, David Gwilliam, Joshua Go, Vijendar Ganta, Bharath Vikrant Iyer, Manish Dhingra, Anon User, Marin Dimitrov, Drew Houston, Gordon Mohr, Andrew de Andrade, Daniel Shi, Chaz Yoon, Ted Suzman, Ashwin Bharambe, Tomislav Bišćan, Nik Black, Nick Perry, Walter Punsapy, Adam Conner, Steven Soneff, Russ Garrett, Sanjay Hora, Garindra Prahandono, Sam Vafaee, Michelle You, Daniel Lucraft, Emiliano Berenbaum, John Song, Joseph Quattrocchi, Bhavin M Turakhia, Sam Parnell, Edmond Lau, Luca Pellicoro, Jeffrey Spehar, Adam Hahn, Robert Shedd, Stanley M Bielski, Juozas Salna, Viraj Mody, Vivekananthan Sivashanmugam, Adam Smith, Teng Siong Ong, Yung-Hui Lim, Edwin Khoo, Ryan Williams, Emmett Shear, Ashish Chordia, Brandon Leonardo, Adam Tait, Ryan Bavetta, Stanislav Vishnevskiy, Kent Beck, Michael Mahemoff, Jade Rubick, Lenny Martin, Andrej Koelewijn, Daniel Ribeiro, Lawrence Sinclair, Shi-Yung Lee, Idoh Gersten, Tuhin Barua, Josh McCormack, Ashit Vora, Alan Chiu, Brian Slesinsky, Luke Melia, Benjamin Kudria, Pat Nakajima, Rob Hunter, Tatum Lade, Adam Keys, Ariel Porath, Andrew Evans, Paul Hamilton, Renaud Guérin, Georg Friedrich, Toby DiPasquale, Rahul Malik, Thomas David Mackey, Sebastian Sanitz, Florian Ebeling, Chris Han, Rob Howard, Roger Hoover, Jon Chiu, Jason Benedict, Lukasz Strzalkowski, Chad Dickerson, Jason Whitman, Scott Hulbert, Gautam Guliani, Dare Obasanjo, Phillip Pearson, Phil Haack, Jovi Kao, Sean Bell, Anthony Bouch, Ghislain Proulx, Andy Hey, Brad Twitty, Lyndon Washington, Michael Gold, Jenna Pederson, Dagfinn Parnas, Adam Caviness, Justin Leitgeb, Cem Unsal, Eddie Kay, Alexander Beletsky, Jeff Woelker, Thomas Wagner, Teresa Thomas, Binh Nguyen, Matt Mercurio, Gaurav Jain, Valentin Bora, Hirak Chatterjee, Ramki Gaddipati, Barnabas Debreczeni, Alexander Zhuravlev, Eugene Otto, Chand Ri, Dain Miller, Marek Foss, Sush Kini, Raymond Zhou, Dan Luu, Ryszard Szopa, Franck Curier, Christopher Meiklejohn, John Bender, Christopher Lin, Guilherme Ambros, Michael Waxman, John Maier, Kai Elmer Sotto, Adam Bossy, David Yongshin Yu, Carlos Leiva Burotto, Max Lin, Rahul Mathur, Amit Chaudhary, Rafał Piekarski, Bob Ippolito and Anton Vinogradenko

We do several kinds of testing. Some specifics:

  • For our PHP code, we have a suite of over 2200 test classes using the PHPUnit framework. They range in complexity from simple true unit tests to large-scale integration tests that hit our production backend services. The PHPUnit tests are run both by developers as part of their workflow and continuously by an automated test runner on dedicated hardware. Our developer tools automatically use code coverage data to run tests that cover the outstanding edits in a developer sandbox, and a report of test results is automatically included in our code review tool when a patch is submitted for review.
  • For browser-based testing of our Web code, we use the Watir framework. We have Watir tests covering a range of the site’s functionality, particularly focused on privacy — there are tons of “user X posts item Y and it should/shouldn’t be visible to user Z” tests at the browser level. (Those privacy rules are, of course, also tested at a lower level, but the privacy implementation being rock-solid is a critical priority and warrants redundant test coverage.)
  • In addition to the fully automated Watir tests, we have semi-automated tests that use Watir so humans can avoid the drudgery of filling out form fields and pressing buttons to get through UI flows, but can still examine what’s going on and validate that things look reasonable.
  • We’re starting to use JSSpec for unit-testing JavaScript code, though that’s still in its early stages at this point.
  • For backend services, we use a variety of test frameworks depending on the specifics of the services. Projects that we release as open source use open-source frameworks like Boost’s test classes or JUnit. Projects that will never be released to the outside world can use those, or can use an internally-developed C++ test framework that integrates tightly with our build system. A few projects use project-specific test harnesses. Most of the backend services are tied into a continuous integration / build system that constantly runs the test suites against the latest source code and reports the results into the results database and the notification system.
  • HipHop has a similar continuous-integration system with the added twist that it not only runs its own unit tests, but also runs all the PHPUnit tests. These results are compared with the results from the same PHP code base run under the plain PHP interpreter to detect any differences in behavior.

Our test infrastructure records results in a database and sends out email notifications on failure with developer-tunable sensitivity (e.g., you can choose to not get a notification unless a test fails continuously for some amount of time, or to be notified the instant a single failure happens.) The user interface for our test result browser is integrated with our bug/task tracking system, making it really easy to associate test failures with open tasks.

A significant fraction of tests are “push-blocking” — that is, a test failure is potential grounds for holding up a release (this is at the discretion of the release engineer who is pushing the code in question out to production, but that person is fully empowered to stop the presses if need be.) Blocking a push is taken very seriously since we pride ourselves on our fast release turnaround time.

My team, Test Engineering, is responsible for building the common infrastructure used by all the above stuff, as well as for maintaining PHPUnit and Watir. Facebook has no dedicated QA team; all Facebook engineers are responsible for writing automated tests for their code and keeping the tests maintained as the underlying code changes.

Facebook’s test setup is still very much a work in progress, but the above is at least a taste of what we do in that area.