With more than 1.5 billion monthly active users (MAUs), Facebook has a lot of factors to take into account for each mobile release. If they followed mobile “best practices,” they’d have some pretty significant problems:
- Deploying to millions of users in one fell swoop (greatly increasing the risk of widespread crashes)
- Asking users to update their apps nearly every week
- Sending unproven features that could tip their KPIs either direction
The goal is to mitigate risk
Different users have different levels of tolerance for experimental features and the inevitable bugs. While users closer to the development of your product such as your coworkers or beta testers may be tolerant of bugs or experimental features, end users are not.
Facebook utilizes their internal feature-flagging tool called Gatekeeper to push out new features and ensure that they’re driving the right KPIs (and not causing crashes). The code looks like this:
This code combined with Gatekeeper allows Facebook to toggle features on and off for any segment of mobile users. They’re able to deploy a new feature to only Zuckerberg’s personal phone, users in Pakistan, or a small percentage of users in California.
This also allows them to break down the release process to gain the maximum amount of qualitative and quantitative data possible, before they roll out to their entire user base. To do so they utilize mobile feature flags to do:
- Dogfooding
- Beta Testing
- A/B Testing
- Field Testing and Staged Rollouts
- Conditional Logic
Let’s break it down.
Dogfooding
“Fail early. Fail often. But fail internally.” — Jason Mark, Fast Company
Dogfooding refers to the process in which a company uses its own product for testing purposes (i.e. eating your own dog food). This process allows them to uncover bugs or usability issues well before any end user sees the product.
Rather than giving employees test devices that they may or may not use on a regular basis, Facebook does something special: They forcibly update employees to early test versions of their apps. Using Gatekeeper, they’re able to enable features only to users who are identified as Facebook employees.
(Related: Putting testing back into DevOps)
Using feature flags for internal testing opens channels for immediate and honest feedback. Employees are quick to share their thoughts and gripes, helping the company validate new changes early on.
The team at Facebook has even built a specific function called “Rage Shake” to help employees report device states during crashes. As the name implies, employees can violently shake their phones to send the device state to Facebook’s QA team.
Beta testing
Moving on, Facebook conducts beta tests to validate it on users with less of a personal stake. With mobile beta testing, users have to opt in, and they tend to be the innovator/early adopter crowd who want to test out the cutting edge. This also means they have a bit of a higher tolerance for bugs and crashes than your typical end user.
While both iOS and Android have their own systems, they’re slow, clunky and hard to use. iOS also comes with a major limitation: a cap of 2,000 beta testers. For Facebook, that’s only 0.00012% of their user base. “This is not enough testers to validate apps broadly at scale,” said Christian Legnitto, a former Facebook engineer. Using feature flagging allows Facebook to surpass this limitation and add users seamlessly by user IDs.
A/B testing
To get quantitative data on how users will behave when shown the new changes, Facebook uses an internal tool called Airlock in conjunction with Gatekeeper that allows them to allocate users for an A/B test.
This allows them to get a different type of feedback: quantitative data on how users are reacting to a new change and whether it’ll help them achieve their goals.
Field testing and staged rollouts
“We often test new experiences with a small percentage of the global community,” an Instagram spokesman told The Verge.
The last stage before a full release is to run field tests, which are experiments conducted under actual use conditions (as opposed to within a lab). In this case, it means that Facebook is deploying its new features to select groups, such as when they rolled out new mobile profiles to users in the U.K. and California, how they did a staged rollout over six months with Timeline, or how they’re testing a new UI for Instagram.
Image via the Verge
Feature flags for Facebook
While most mobile teams deploy to hundreds of thousands of users in one fell swoop, Facebook has done a superb job of creating a methodology that allows them to release validated and tested features out to their users. With their amalgam of dogfooding, beta testing, A/B testing and field tests, they’re able to meticulously segment and glean qualitative and quantitative data from different personas on the tolerance scale.
If mobile is to advance as an industry, we need to stop waiting on Apple to make the necessary changes to the App Store and start taking back the control. By either building or buying A/B testing and feature flagging solutions for mobile, more teams will be able to closely monitor deployment, validate features early on, and release on their own terms.
