In January 2012, three Twitter engineers were in charge of writing and maintaining the entire codebase for Android. New versions of the app were released every two to three months. Twitter’s mobile development needed to scale way up, and to do that the social networking company needed not only to restructure its development team, but also to change its corporate culture.

Twitter for Android engineering manager Jan Chong spoke at the Velocity Web Operations and Performance conference in New York City. Her presentation on “Scaling Android Development” talked about the road from a single team-based development model to a project-based model across distributed development teams. The first thing Twitter had to do, she said, was to break away from its fixed way of thinking, and start thinking about development in a mobile-first context.

(Related: Rounding up the big news and keynotes at Velocity NYC)

“Twitter had Web Brain,” Chong said. “It came of age on the Web and had its initial growth in this area, which provided a lot of challenges when we wanted to go mobile.”

Shifting to mobile-first, Android-first
Twitter for Android began in 2011 with three developers and one designer, including Chong. After convincing the company’s executive board of the need for a more robust mobile presence, she was able to grow the app team from three to a dozen over the next year and a half, accelerating the app’s release cadence from every 8-12 weeks to once a month, then bi-monthly.

In shifting from a Web-first to a mobile-first philosophy, Chong and the Twitter for Android team had to nix Twitter’s #ShipIT concept, getting a release out as fast as possible. One complication of developing for Android as opposed to iOS is that users choose when to install an upgrade. On the Web, an inexperienced developer can code functionality and bugs aren’t as big a big deal, she explained. On Android, bugs can live a lot longer out in the wild, which has implications for API design. While in 2014 more than 80% of the Twitter for Android user base is running Jelly Bean or above, that still leaves several different operating systems to ship upgrades for on a weekly basis, before even factoring in device fragmentation.

“After you release a version, be prepared to support it for two years to avoid cutting off active users,” Chong said.

A constant state of flux
In the past nine months, the Twitter for Android team has grown rapidly from a staff of twelve to dozens, with 12-15 independent product groups including the frameworks team, the UI team, and the performance and stability team. The app has also moved to a weekly release cadence where two releases are always staggered: one in active development and one in bug fixes, with a mostly automated “bake” at the end and a new release branch every Friday.

Another critical process the team added is feature switches, allowing developers to dynamically enable and disable features on the server-side so they can deal with bugs without pushing a new version or a hot fix. It also allows teams on different schedules to both work on the app’s master version without holding each other back.

Chong and her team are constantly evaluating and changing how they do things. The fragmented device landscape ensures that every three to six months, she revises her device popularity rankings by market, adding and removing devices from the list, each with its own unique hardware and screen size challenges. The team is also considering changes in its device lab, tilting more toward rapid and consistent feedback as opposed to its current on-device unit and functional testing.

Diving into Twitter for Android’s guts, Chong revealed what software the app uses to accomplish various tasks, and the growing number of open-source technologies being integrated and replacing one another as the platform evolves.

Twitter for Android uses the uiautomator framework for UI testing, Git for source control, and Jenkins for Continuous Integration. For build automation, the team transitioned from Apache Ant to Gradle earlier this year, and they’re in the process of moving from Robotium to Espresso for user scenario testing.

Principles of an Android Brain
Chong preached the concept of transitioning a developer’s way of thinking from a Web Brain or iOS Brain to an Android Brain. The mobile platform is more fragmented, customizable and widespread than any other. As such, she laid out several Android-specific principles of development.
• Reliable, short release cadences: Twitter for Android increased its release cadence from quarterly to monthly to weekly, speeding up release trains to “make it cheaper to take the next train,” said Chong. In doing so, it spread changes over time rather than stack them in one bug-ridden release.
• Experimentation is a first-class citizen: In terms of processes, tooling and support, a mobile experimentation framework allows developers to work kinks out and try new things, solving riddles like the varying user experiences from mobile Web to native, especially with what Chong described as an “immature” IDE in Android Studio.
• Buy a lot of devices: Every Twitter for Android developer has a stack of phones on their desks, and now the entire team—engineering and product managers, designers, even executives—uses an Android device at work, according to Chong.
• Eat your own dog food: Employees use internal builds (also known as dogfood builds) to compare directly against production, see where bugs are, and tweak from their phones. Everywhere inside the build, developers can automatically report a bug with a screenshot and device info, automatically creating a ticket. The dogfood build occurs automatically at 3 a.m., stopping developers from slowing down the build by adding features at the last minute.
• Constant monitoring: Twitter for Android uses Twitter’s free Crashlytics product for real-time crash tracking, as well as stability and configuration management. The platform and QA teams work closely on critical bug fixes.
• Rapid testing and feedback: The app undergoes per-commit user tests, functional tests every few hours on every supported OS level, and longer nightly tests on upgrades and other complex features. Dedicated users in the Google Play Alpha and Google Play Beta channels provide real-time feedback on new features and versions.
• Send developers to class: Developing for Android is unlike any other mobile platform, so treat it that way. Twitter for Android now runs its own Android development class, putting each newly hired developer through training.
• #AutomateIt: The team laid out a set of style rules to enforce automatically within Lint error-flagging tools, and every style change outside of these rules is purely optional.