The year was 1997. Steve Jobs fidgeted on a stool in front of the World Wide Developer Conference, chatting with the audience: “You’ve got to start with the customer experience and work backwards to the technology. You can’t start with the technology and try to figure where you’re going to sell it. I’ve made this mistake probably more than anybody else in this room — and I’ve got the scar tissue to prove it.”
Jobs was deftly answering a man who had just accused him of abandoning a pet technology. The Apple founder went on to explain that his company’s mission was to discover “What incredible benefits can we give to the customer? Where can we take the customer? Not, ‘Let’s sit down with the engineers and figure out what awesome technology we have and then how we’re going to market that.’ And I think that’s the right path to take.”
As it happens, 1997 was also the year Jony Ive became Apple’s senior vice president of industrial design. He went on to determine the curves, gloss and heft of the iMac, iPhone, iPad and more. Two years ago, the San Francisco-based Ive became Apple’s chief design officer — a role that exemplifies how strategic user experience has become in the technology world.
Inclusive design is strategic design
Unveiling his third annual “Design in Tech Report” in a March 12, 2017, talk, Silicon Valley design guru John Maeda noted that design is now a top priority for venture capitalists, consultancies and even stuffy enterprise software giants: “IBM design has been probably the largest corporate effort to amass design energy. […] Google is cool. Who would have thought? The perception on Google [has] definitely shifted.”
Google has indeed changed its tune. Google Design has created “a visual language for our users that synthesizes the classic principles of good design with the innovation and possibility of technology and science,” according to the spec at Material.io. The Material tools and components help developers build mobile-ready cross-platform experiences that have touch, voice, mouse, and keyboard as first-class input methods.
In Maeda’s formulation, “computational design” is a discipline that melds artistry, business, engineering — and inclusion.
“In my official title at Automattic, I’m the global head of computational design and inclusion. People ask me, ‘Why do you have the word ‘inclusion’ in your title?’ It’s because I believe that design and inclusion are inseparable,” he said. Creativity is intrinsic to inclusion, according to Maeda, but that energy is lost when inclusion is relegated to a human resources process rather than seen as fueling beautiful user experiences.
Inclusiveness is a common theme for Google as well. Reaching “the next billion users” was a mantra at the Google I/O conference in May 2017. Google speakers noted that many of these future customers are now or will be disabled: one in five people will have a disability of some sort in their lifetime.
“This isn’t just for users with a disability or an accessibility need. I want to get across that this helps all users,” said Patrick Clary, a product manager on accessibility at Google who himself uses a wheelchair, in his Google I/O ’17 talk, “What’s New in Android Accessibility.”
Why should accessibility interest app developers? Blind or low-vision products can help those who have their eyes otherwise occupied, such as drivers, he said. Designing for those with motor impairment helps others who can’t use their touch screen because it’s inconvenient or dangerous. “It’s really about designing for the widest possible range of abilities within the widest possible range of situations,” Clary said.
Android accessibility settings, APIs and long-running services are nifty developer tools for changing how users consume or interact with devices. For blind users, services include TalkBack and BrailleBack (which can activate a refreshable braille display), while Switch Access and Voice Access are targeted to those with motor impairment such as a tremor.
Meanwhile, Apple’s design aesthetic continues to revolve around user experience. At the 2017 WWDC, the company reminded attendees to develop not for “users”, but for humans. It turns out this is a longstanding tenet for the company: Apple’s evolving Human Interface Guidelines actually date back to 1987, which was also the year the Macintosh II personal computer was launched. At this year’s WWDC, the company maintained a forward view with its emphasis on humanity — and not just what humans see and do, but what they hear.
Sound: The next frontier
In 2003, Web usability expert Jakob Nielsen wrote, “Visual interfaces are inherently superior to auditory interfaces for many tasks. The Star Trek fantasy of speaking to your computer is not the most fruitful path to usable systems.” He was wrong.
With Siri, the first commercially viable personal assistant, the 1970s futurama had come to fruition. By 2020, Gartner predicts that nearly a third of all web browsing will be done without a screen and 85% of customer interactions will be managed by bots. ComScore predicts that half of all searches will be via voice. Sound: the next frontier.
According to Apple sound designer Hugo Verweij, sound can transform user experience, but too often, app developers miss the opportunity to compose custom audio notifications to distinguish their apps from others. In a compelling talk at WWDC 2017, Verweij offered important guidelines for using sound: “Will my app send frequent notifications? Can sound play a role in my app’s branding? Can the UI benefit from an audible component? How would I understand my app without a GUI?”
“Don’t overdo it — silence is golden,” he said, displaying a hilarious cautionary example of the iOS maps app overdone with silly sound effects, as if the comedian Victor Borge, of the famous “Phonetic Punctuation” routines, had commissioned it.
“If you’re making a game, it makes sense to make a whole world of sound, but we don’t want every app to sound like a game,” he said, noting the importance of always giving users the option to mute apps as well.
When it comes to sound, details matter. It can be a tricky game of trial and error to synchronize sound to haptics or animation — and getting it wrong can create illusions such as making buttons feel sluggish, or awkwardness when sound isn’t synchronized to video. When it comes to editing, while it’s advisable to work with an expert sound designer or sound engineer, simple tools such as Garage Band can make a huge improvement, Verweij advised.
Getting back to the Star Trek scenario, a plethora of machine learning APIs from Google (Cloud Speech API, Cloud Natural Language API), IBM (Watson Conversation), Oracle (Chatbots) and more make it easier than ever to harness voice interfaces for new apps. As it happens, that same combination of artificial intelligence and big data that’s powering machine translation and speech also holds the promise for predictive user experience.
Predictive UX with AI and analytics
One thing the design-focused Jobs might not have foreseen is how much user data we would be accumulating in 2017 — and how rapidly we are learning to put it to good use. User experience is no exception.
“In the UX world, AI and automation is transforming the role of the designer. Traditionally, UX teams would turn to metrics and tools such as usability tests, usage data and heat maps, to understand how to improve the functionality and effectiveness of a system. However, in the age of AI, we now have empirical, actionable data that we’ve never been privy to before, giving us greater granularity into optimizing the user experience,” said Rephael Sweary, cofounder of the San Francisco-based digital adoption platform WalkMe.
According to Sweary, AI helps conduct quantitative usability testing, easily extrapolating characteristics such as:
- Location, job title, device
- Time of day and length of session
- User flow and drop rates within the application
- Behavior analysis based on screen recordings of drops from user flows
- Total number of users, unique visitors and sessions
“What we do with AI is optimize adoption. We define a goal for the AI algorithm, like ‘increase users who use feature X.’ We run our AI algorithm across our entire data set and look for people who use this feature. Then we predict adoption based on the people who use this feature. For example, people who use the app more than three times at work uploading two or more photos are most likely to use the ‘share’ feature,” said Kobi Stok, director of mobile product and technology at WalkMe. The company calls the ideal time to make a request or introduce a feature the “happy moment” for user engagement.
User experience metrics can also improve onboarding and training. “Typically, training is done with a firehose approach. You take a group of users away from a productive line of work, you train them for a few days and then you send them back. Wouldn’t it be nice to tailor training to only issues they have been experiencing while using the software?” asked Bogdan Nica, vice president of product and services for Knoa Software in New York City.
Knoa Software specializes in SAP application performance management. Now, as SAP is consolidating around a user interface revamp called Fiori, Knoa’s UX metrics can help ease the migration and identify “adoption gaps”.
“A main pain point that SAP users have had is that there are so many different UI standards,” Nica said. “A major migration takes a year. It makes sense to start collecting data before the migration to establish a baseline. You continue collecting during the migration. Then, when it’s completed, you take a look at metrics at the end of project so you can do a before-and-after analysis, but also to make sure it’s fully adopted — to identify adoption gaps, because there’s always something that goes wrong. Maybe everything works from technical point of view, but business processes are out of whack.”
Enterprise software design is being forced to improve user experience as it competes with consumer apps for employee attention, Nica notes. “There are different expectations of what good software looks like now. You can no longer force customers to use business software that looks like it was designed in the 80s or 90s,” Nica said. But what about apps that are too immersive? As design grows in importance, so does the obligation to use it responsibly.
Thinking about designing user experience responsibly should join security and privacy as a first-class concern — and it has become a priority to limit, say, texting and driving through driving detection in mobile apps. Like security and the other “-ilities,” it may still get lost in the shuffle as developers strive for faster releases. Unless… Could a new class of hybrid UX/techie bring these issues to the fore?
The new designer/developers
As a profession, design is embracing software development technology, Maeda believes. His surveys find more and more hybrid designers who have coding skills in JavaScript, PHP, or Ruby on Rails. He emphasizes that computational design requires an ability to iterate based on UX metrics, understand algorithms and embrace cutting-edge form factors such as self-driving cars and other connected devices.
And some hybrid designer/developers, like Michael Hoffer, started on the developer side. He’s a research scientist at Goethe University in Frankfurt, Germany, who created VRL Studio, a slick visual programming environment for Java.
“There are many powerful textual programming languages out there that already have a diverse and comprehensive ecosystem around them. Building a new visual programming language is challenging, at least if it is supposed to serve as a replacement for general purpose programming languages. For me, it is very important to provide visual programming environments that do not isolate developers from the ecosystem of existing languages and platforms. Therefore, I develop new interactive visual representations for existing textual programming languages,” said Hoffer.
VRL is not only sleek and powerful, it’s easy on the eyes — and Hoffer has done this on purpose: “Aesthetic aspects play a huge role. Actually, they are important for textual programming languages as well. Even though outsiders do not usually understand the beauty of well-structured source code, developers who have to look at and work with that code all day long do certainly develop a taste for beautiful code.”
The same applies for IDEs, Hoffer believes, and it can help developers find productive flow — and even reason more effectively about program structure: “Providing a good user experience for developers is highly important. Designing development environments, especially visual programming environments, that are aesthetically pleasing is very hard. Good user experience is correlated to finding the right abstractions. For general purpose development environments, this is especially hard because any simplification runs into the danger of limiting the possibilities of the IDE.”
As software becomes ubiquitous, much of its arcana will be made accessible to the masses via more beautiful, inclusive and usable designs — a fact that has motivated SAP founder Hasso Plattner to fund prestigious design schools around the world.
“Hasso Plattner is a very systems-oriented guy. He’s the architect behind the HANA in-memory database technology, but in a lot of the recent events that he’s had, he’s started to focus more and more on the UX side,” said Knoa Software’s Nica. “They realize no matter how powerful SAP is on database or server side, none of that matters if users can’t use it. That includes AI, moving to cloud — if that is not done with the ultimate objective of improving the user experience, none of that matters. It’s a validation that you cannot fail in software if you singlehandedly focus on the user. That’s your best course of action.”
Design education: Learn more
If you want to learn design, there are a growing variety of options, starting with the written word. In Make it New: The History of Silicon Valley Design (MIT Press, 2015) Barry Katz spotlights how influential design has been since tech’s early days.
John Maeda’s annual “Design in Tech” report, now in its third year, provides an invaluable snapshot of industry trends. Online resources for insights, education and training include free and paid blogs and courses at MIT Media Lab, Design.blog, Wizeline, Lynda.com, Youtube and Pluralsight.
There are also brick-and-mortar schools: You can get an MFA in interaction design from New York City’s School of Visual Arts, attend the Center Centre (formerly the Unicorn Institute) in downtown Chattanooga, TN, for its two-year user experience design program, or get a BFA in UX from SCAD (Savannah College of Art and Design, in Savannah, GA). Finally, around the world, three Hasso Plattner Institutes of Design Thinking have sprung up thanks to SAP founder Plattner’s philanthropy. These “d-schools” are sited at Stanford University, Potsdam University and the University of Cape Town.
