Want to push out software updates automatically to your customers? Don’t do it! Even if you have tested, tested, tested and then tested some more, better think twice—especially if your software goes into an embedded system. Let me explain why.
The weather in much of the United States has been chilly for the past few weeks. For owners of some Nest Learning Thermostats, it’s been chilly indoors as well.
The Nest Learning Thermostat is a very clever, albeit expensive device. Hook it up to your furnace or air conditioner, and not only does it learn what temperatures you like, but it also connects to your home or office WiFi network. That lets you monitor and control it remotely from a browser or mobile device. If you have multiple Nest devices, they will talk to each other.
We have two Nest thermostats in our dual-zoned home, and they are swell toys. Have they saved us money by optimizing our heat and air conditioning? No idea. Do they justify their US$249-per-unit price point? We doubt it.
What the Nest definitely can do, however, is fail if the company pushes out a buggy software update, which happened in late November. The Nest 4.0 firmware apparently caused some number of Nest devices to go offline, leaving folks without heating or cooling.
Fortunately, my family’s two Nest devices upgraded without a hitch. That didn’t happen with my friend Michael Miller—read his justifiably unhappy comments on his ForwardThinking blog post, “The Nest Learning Thermostat and Why the Smart Home Is Pretty Dumb.” Michael writes:
But the real issue came last night when suddenly the display went black except for a blinking green light on the top. I surfed the support pages on the Web and then called technical support. There I got a message that the company had pushed out version 4.0 of the Nest software, which had caused some thermostats to go offline. It went on to say the company was working on rolling back to an earlier version, and that process “should take a few days to complete.” According to the company, that should fix the immediate problem, but in the meantime you could try updating your router’s firmware. I tried waiting for a support technician, but the phone message said they were understaffed and hoping to hire more people soon. Left unsaid was why the company would push out such a software update without informing or asking the users.
Maybe in more temperate Silicon Valley, where Nest is headquartered, a few days without heat or cooling is acceptable. But in the Northeast where I live, it’s a potential disaster. It was 18 degrees Fahrenheit outside and because the thermostat was offline, we had no heat. Thankfully, the electrician was able to come over around midnight, pull out the Nest, and install a more standard thermostat. It’s not particularly “smart,” and I can’t control it from my iPhone, but it keeps the house warm and that’s more important.
The problem is still not solved. In early December, a user, drivingmissm, posted on Hacker News under the heading, “Tell HN: Nest Software Update Failed and Now We’re Freezing.”
I have a Nest learning thermostat and a house with two infants. We’re on the east coast and the outside temperature is 26 degrees with seven inches of snow on the ground. Several hours ago, this evening, our Nest began cycling “software updates” that turned off our heatpump. Nest support is overwhelmed—apparently this is a system-wide problem related to an OS update that failed. We’re freezing and are getting ready to go to a hotel.
The “Internet of Things” needs different standards than some Internet app—this software failure is disrupting lives.
And then added:
Update: Just got through to tech, they are working late. Apparently our issue is an edge case but there are a bunch of problems with their OS update and automatic rollback. Here’s the temporary fix: To reboot a Nest, press it for 10 or 15 seconds. Then, when the Nest is booting up, go into Settings and disconnect the Nest from the Internet so it doesn’t try to software update endlessly. That has worked and we thankfully have heat again. I generally like my Nest and have found their support to be excellent, so I’m pretty patient, but it is definitely exasperating to hold a crying baby while calling tech support in the middle of the night because of a failed software update on my thermostat!
The point here isn’t to pick on Nest. It’s to point out the hazards of pushing out software updates, especially to real-world embedded systems. Sure, it’s a pain if our phone app crashes due to an update, or if we have trouble starting a desktop app. However, in our increasingly connected world and the trend toward the Internet of Things, we developers simply should not place our customers in this situation.
Turn off automated updates. Software updates should be opt-in every time. Every time.
Alan Zeichick, founding editor of SD Times, is principal analyst of Camden Associates.