Category Archives: device

Descriptions of devices and what it would mean to make them end-user-programmable.

Bathroom Humor

New York Times blogger Nick Bilton writes, in “Disruptions: As New Targets for Hackers, Your Car and Your House“, that researchers at the DefCon conference reported hacking into hybrid cars and aggressively influencing their behavior.  The blog says

The researchers completely disabled a driver’s ability to control a vehicle. No brakes. Distorted steering. All with a click of a button. 

Eek.  Even more vivid descriptions can be found on the Forbes.com site by a journalist who was driving a hacked car and experienced the effects first hand.  It’s definitely frightening, but strikes me as a little overblown.  The current round of hacks require that the hacker have a device physically connected to the diagnostic port on the car, so it’s not like any old baddy with a smartphone can crash your car.  On the other hand, wireless attacks on cars have been reported in the past.  Colleagues in my former department showed how to mess up a car by communicating with its wireless tire sensors from bluetooth transmitters by the side of the road.  So, while people are not reporting that any of these life-threatening interventions are happening in the real world, it should give one pause.

 

In addition to sounding the alarm about breaking into cars over the Internet, the article cites some risks of using Lockitron, an online door lock that I have on order, online lighting like the Hue I have, and Internet-controlled refrigerators.  There’s a real concern that Internet-enabled user-programmable devices of the kind I advocate could open us up to some serious malicious mischief if designers don’t keep security at the forefront of system design.

The line that convinced me it was worth making my first blog post in many weeks, however, was in the context of the INAX wireless toilet.  This state-of-the-art commode sports, not one, but four automatic features.  Here’s the description from their site:

  1. Automatic Lid & Heated Seat: When you approach the toilet, the lid opens and the heated seat is activated.
  2. Sound Module: When the lid automatically opens, music from the sound card will begin to play and the deodorizer will be activated.
  3. Automatic Flushing & Deodorizing: When you step away from the toilet, it will flush automatically.
  4. Self-Closing Lid: When you are finished, the lid closes automatically, the deodorizer deactivates and the air purifier will activate emitting ions to cleanse the air in the room surrounding the bowl.

Actually, that sounds kind of awesome.  Maybe not worth the $5600 price tag, though.  And it’s even more expensive if you want to control these features, as there is a $500 remote control center that they offer.  Happily, these features can also be controlled from your smartphone! Unhappily, the bluetooth security code built into the toilet is, by default, set to ‘0000’.  So, anyone with the free app can control any toilets in the vicinity.  Oh, ick.  Now, although I wasn’t able to verify it, the Times blog post claims that the company has issued a patch, and summarized the situation with this gem:

Yes, in the future, you will need to download security updates for your toilet.

That cracked me up.

Unfortunately, in the process of researching the topic for (I thought) a quick blog post, I ran across an article that envisions what happens when homes are only as secure as the passwords people use to secure their website access.  In short, it summarizes the various physical items that are being put online and pairs them with what we know about the history of attempts to secure websites, which hasn’t been stellar.  One of this posts concluding thoughts about the security situation for controllable devices:

The rush for vendors to compete in this extremely fast moving market will inevitably result in rushing aspects of the product design and we know very well from past incidents that security is one of the areas most frequently overlooked in favour of delivering features.

Ok, that makes me a bit paranoid.  As more stuff goes online, developers and end-users will get sloppy and opportunities for troublemakers to hurt people and property expand tremendously.  It seems kind of inevitable.

Although I never watched Battlestar Gallactica, I saw most of the pilot.  A critical plot point is that all the ships in the defense fleet have all their weapons disabled by a backdoor in the control system software installed by the enemy.  The only battlestar that survives the vicious attack is the one with the paranoid captain who refused to trick out his ship with the latest in software control.

Boy, I sure hope there’s another choice between refusing to build programmable systems and getting exterminated.  I would really like to have my programmable devices and my life. Perhaps there’s a role for universities to create a lightweight free open security package that could be offered to developers everywhere?  Maybe, if it’s easy enough, consumers will demand a certain minimal level of security in the products they buy?  Perchance such developments might help stave off the doomsday scenario?  Help?

Wired’s Programmable World

Wired magazine has an article on what they call the “Programmable World”.  (Thanks for the pointer, Blase!).  I feel like that’s a pretty good description of the world I want to live in, so I read the article: http://www.wired.com/gadgetlab/2013/05/internet-of-things/all .

Here’s a short reaction to it.  I liked it.  Here’s a (slightly) longer reaction.

The article asserts that the programmable world will arrive in three stages:

1. More and more devices get onto the network.

2. These devices work together to automate tasks.

3. The devices are organized into a programmable software platform that can run apps.

I suppose I agree with this sequence in terms of the underlying technology, but—as is often the case—the evolution of the technology is less interesting to end users than the evolution of the end user experience.  Here’s my take on what the transformation could look like to regular folks:

1. More and more devices become available with their own iphone/android apps.  That’s a safe and easy prediction because it’s already well under way.  I just bought a Jawbone Up (an activity monitor), for example, and it comes with a free iphone app for plotting daily activity and sleep cycles.

2. Cross-device interfaces like IFTTT or Smartthings (which I’ve written about in the past) will make it possible for end-users to specify simple programs that involve multiple devices.  I’m not as sure about this prediction.  End user programming may or may not catch on (although I deeply believe it should).  Regardless of how popular they will become, these cross-devices interfaces are already here.  In fact, I bought the Up because I have a rule that says “if a device is advertised as working with IFTTT, I must buy it.”  So, I have a little iFTTT program now that turns on colored lights (Phillips Hue, which also just got integrated with IFTTT) in my apartment if I reach my per-day step goal.  It’s not at all useful and it doesn’t work the way I want it to.  But, it’s cool that it’s becoming possible to knit these gizmos together at all.

3. We move from a sensor-action model to an event-activity model.  I’m not really seeing this stage in the real world yet, but the more I play with programmable devices, the more I think the whole enterprise flounders without it.  Here’s a concrete example.  The Wired article talks about  automating our daily tasks.  They mention that a computational system should be able to observe the rules you the user follow and then adopt them itself.  So, if you behave according to the rule “If  the sun hits your computer screen, then you lower a shade,” then the computer could do it for you and save you a step. That’s fine, but here’s the thing.  That’s not really the rule we follow.  It’s something more akin to “if I need to see the screen and I can’t, do something to make the screen visible.”  There is no sensor that can directly measure whether you can see the screen.  Such an event needs to be inferred from a set of observables like the brightness of the screen, the direction you are looking, how long it has been since the mouse was moved on the computer, the time of day, whether the shades have already been pulled, etc.  And the action of pulling the shades isn’t always the right response.  It might be more effective to brighten the screen, rotate it a little, or perhaps dim the lights in the room.  We need a computing infrastructure that (1) can integrate information across devices, (2) infer the desirability of possible interventions, and (3) orchestrate a coordinated response across devices.

From the end user’s perspective, this last step requires careful interaction with the devices.  They need to learn from you without distracting you.  The line between teaching, training, and programming begins to blur as we tell the computing systems what matters to us, how it can be measured, and what to do in response to undesirable situations.

Well, that’s the trajectory, anyway.  For now, the best bet is to focus on getting an infrastructure in place so we can start experimenting with different models of interaction.  We have lots of goals for this summer and I hope to be able to report on exciting and steady progress over the next few months!

Some things About SmartThings

It’s been a busy semester and I haven’t been able to find time to write.  And the longer I wait, the more I feel like the next post should be really good to justify the gap.  But, now it’s Spring Break and maybe I should just write something bad just to take the pressure off.  🙂  Here goes.

Yesterday, I was fortunate to have a visit from the former CTO of iRobot, the makers of the Roomba vacuum cleaning robot and perhaps history’s only profitable robotics company.  Tom Wagner stopped by mainly to talk about the Human-Robot Interaction initiative my colleagues at Brown and I are trying to sell.  But, since he was there, I took the opportunity to tell him a bit about the end-user programmable devices project.  Blase Ur participated from Pittsburgh via a robotic telepresence device called a Vgo.

vgo

We demoed several programmable devices—a fan, a blender, and two strings of colored lights—and asked Tom for feedback.  He was amazingly insightful and refreshingly forthright on a number of issues.  Unfortunately, many of the juiciest tidbits he shared were prefaced with “Don’t blog this, but…”.  Oh, well.  But, one excellent comment he made that got us thinking was “How does your project relate to SmartThings?”

Mmmm.  Well, I didn’t recognize the name at first, but SmartThings is a project that Vukosi Marivate brought to my attention as a great example of work relevant to end-user programming that was seeking funding through KickStarter.  I probably should have followed up right away when he first told me, but I always feel very much teased by KickStarter campaigns.  The pleas for funds are very compelling, but so many of the amazing ideas never materialize that it’s often a big letdown.

Briefly, SmartThings is a venture to create a networking infrastructure for programmable physical devices.  In the words of their homepage, “SmartThings adds intelligence to everyday things in your world, so that your life can be more awesome.”  Huh.  Awesome is good.  They’ve got over a million dollars in funding through KickStarter and three million in seed funding.  They’ve got over 30 people working on programmable devices.

Their vision is to have a platform so that developers can create apps that interface with physical devices.  They’ve got a cool hub that all the various devices in your home can talk to.

hub

The hub acts as a relay station for messages between the devices and the SmartThings cloud server.  Then, you can create apps that communicate with the cloud to monitor and control the devices.  It’s even possible to reconfigure and automate the behavior of the devices.  In addition to their own custom devices, they are hard at work making other wifi devices compatible with their system.  All the devices I’ve bought for my apartment in the last few months appear in their preliminary list of compatible devices, which is very encouraging!

I think SmartThings is an exciting development, but it’s hard not to feel a little competitive with them.  They did a demo of their system January 2013 that features the creation of an app that monitors a switch to turn a light on and off.  The audience loved it.  But, our upod project demoed precisely that same functionality (and more!) in November 2011.  We had a cloud server, a switch, lights, and a programming interface.  We didn’t have dozens of people and millions of dollars.  We had a handful of undergraduates and a few thousand dollars for parts.

For me, the frustrating part is that nothing like SmartThing existed back in 2010 when we started.  If it had, we wouldn’t have put the energy into creating our infrastructure and would have used theirs.  But, we couldn’t use theirs because it didn’t exist.

Based on their demo, it appears that they have two programming interfaces for their system.  One is very very close to the When-do interface we built for upod (which is, itself, inspired by the if-this-then-that design).  In our experience, this interface style is extremely easy to use, but also quite limited in its ability to express anything beyond the most basic programming functionality.  Their second programming interface is essentially raw code—lots of keywords and curly braces and colons.  It provides much more complete functionality but at the expense that non computer scientists would find it impenetrable.

In the upod project, we experiments with a kind of middle ground, which is a fully functioning programming language that is friendly and easy to use.  Specifically, we extended Scratch with blocks for listening to and controlling devices.

Looking ahead, we think there’s a lot more that needs to happen to make device programming friendly for the end user.  In particular, real-world devices get their information from real-world sensors.  Whereas if-this-then-that has recipes that can be specified with absolute precision like “When a new book is added to Kindle Top 100 Free eBooks, send me an email”, real-world sensors are more vague and messy.  Consider a recipe that appears in a screen shot in SmartThings’ advertising video: “when it’s going to rain, check if Barkley is in the yard”.

smartthings3

smartthings1 (1)

 

It’s a perfectly reasonable recipe and exactly what you’d want to be able to say as an end user.  But, questions like whether it’s going to rain or if a dog is outside hide a tremendous amount of complexity about how information is gathered from the physical world.  I think it’s terribly unlikely that SmartThings has this functionality working in a way an end-user can leverage yet.  But, it’s a lovely and important research question.  And, it’s what I’m anxious to work on assuming SmartThings really has solved the infrastructure problem.  I can’t wait until summer!

 

News Roundup

I love to get pointers to interesting relevant stuff going on in the world at large.  Here are a few links I was sent recently.

Ari Weinstein shared the news that Belkin announces WeMo Smart, coming to your coffee pot later this year.  I haven’t talked about WeMo yet on the blog, but it’s a gadget that I use in my apartment to control my lights.  It connects to my local wifi network and lets me use an iPad app or even IFTTT to write simple programs.  They sell an outlet and a motion sensor, so one program I have turns on my bedroom light for a minute in the evening whenever motion is detected.  I’m a fan of wifi-connected devices, so I appreciate Belkin’s existing product.  The new product isn’t described in any detail in the press release, but, reading between the lines, I think the idea is that they will produce a piece of electronics that will be embedded in the coffeemaker so it can be controlled and monitored via the WeMo cloud.   I don’t know if Belkin’s protocol is open, but they were at least willing to team up with IFTTT, so I’m encouraged by this development!  (Neither Belkin nor IFTTT has embraced general programming as an interface, so there’s still work to be done here.)

I also like the model that both Belkin and IFTTT have used—don’t solve all possible problems up front.  Instead, offer something small and simple that works and then grow from there.  Had Belkin attempted to tackle “home automation” in one go, they would have failed.  If IFTTT had attempted to tackle “end-user programming” in one go, they would have failed.  I very much like the way they are sneaking up on their targets.

The next item, sent by my colleague Shriram Krishnamurthi to the Brown CS faculty, is tangentially related: Ford, GM roll out red carpet for app developers.  Apparently, high end Ford and GM cars run a user-facing computerized system called Sync that interfaces with smartphones.  The news is that they are opening up the system to third-party developers to create car-based apps.  As an example, the article mentions one that reads newspaper articles out loud to the driver.  They are planning to prohibit software that will distract drivers with video, games, and the like.

According to Wired’s Exclusive: Ford Wants to Create the Android of Automotive Apps, they will also go further and offer their platform to other automakers; the idea being that the existence of a standardized platform will make it easier for software developers to invest the time needed to create outstanding applications.  One touchy point, apparently, is whether the system will be integrated with the car’s functioning (beyond the screen and audio).  The more integrated it is, the cooler the apps can be, but the less likely competing car manufacturers will be to trust it.  Plus, I suppose it could be quite risky to allow an app to mess with steering, braking, or even headlights and such.

For what it’s worth, I’ll note that the move to open up the platform to third party developers is far from inevitable.  I was just getting excited about the apps available for the Livescribe computerized ballpoint pen.  (The Zork port is pretty neat and the demo where you draw a piano and then you can play it with the pen is a kick!)  I was hoping students might be interested in developing software that can use the pen as an interface to household devices and then I discovered a page proclaiming Livescribe Discontinuing Developer Program.  That’s so sad and it seems like precisely the wrong direction.  We want to talk to our machines!

Partner-in-crime Blase Ur sent me Computer security and the modern home, an academic paper that analyzes the vulnerabilities of the collection of networked, computerized devices that are becoming available.  Just as programmable devices bring the flexibility of software into the physical world, they also amplify the opportunities for software to enable physical crimes.  The paper  provides a thorough, if dry, analysis of what makes computerized home devices risky from a security standpoint.  They did use one wry phrase I particularly liked, though.  The authors were describing the risks inherent in the fact that home devices are not generally administered by professionals.  Toys, in particular, may end up being particular problematic because they are effectively given to kids to administer

…despite the child’s likely lack of experience with computer security and different stance on privacy issues.

I assume the authors’ experiences with kids’ “privacy stances” are similar to mine—kids are willing to say just about anything to just about anyone.  Even in the unlikely case they understand how to keep their electronic toys from violating the privacy of people in the home, they wouldn’t understand why.

I respect the authors’ efforts in highlighting security concerns.  I recognize the importance of securing our homes and the risks that come with extending global communications networks inside them.  Nevertheless, reading about all the horrible things that can go wrong makes me feel all icky.

Lastly, the story Welcome to the home of the future sounds like it should be relevant; it’s pretty close to my first blog post title, in fact.  I didn’t find it interesting, though.  It talks about the excitement at companies like Apple and Microsoft about the market for advanced household devices, which is is a good thing.  But it focuses on things that are weirdly futuristic and doesn’t really tell a convincing story of how we might get there.  On the positive side, I lost the information about who sent me the link, so no one needs to feel bad about it.

Survey of Programmable Appliances

My motivation for creating this blog was to convince you of the importance of being able to talk to your machines by programming them.  With that goal in mind, I thought it might be worthwhile to take a look at what sort of devices are already considered “programmable” and what people think that means.

Here’s what I did.  I went to amazon.com and searched for the term “programmable” in their Appliances section.  If my calculations are correct, Amazon sells 231,647 products in its Appliances section, 452 of which (a mere 0.2%) use the word “programmable” in the product name.

I organized the products into 18 categories ranging from “air conditioner” to “wall control” and counted up how many appeared in each category.  One thing I found surprising is that coffeemakers make up more than half of all programmable appliances on amazon.com.  On further reflection, this discovery is consistent with my theory that our nemeses in the coming robot apocalypse really just want to make us breakfast.

mrcoffee

To get a sense of what these products do, I hunted down user manuals for at least one in each category.  Based on this documentation, I was able to further group products by what “programmable” means for each.  Aside from one that I couldn’t figure out why it was called programmable (a Meiko dishwasher with its “award-winning MIKE 2 controller” ) and one that I couldn’t get any sense of what the product was intended to do (“GE Part Number RAK148P1 T-Stat 2 Stage H-Pump (Programmable)”), their programming functionality fit into five classes:

  1. Timers
  2. ‘Stats
  3. Schedules
  4. Scripts
  5. Presets

Here’s what they do.

Timers

For the vast majority of the programmable devices—by which I mean the coffeemakers—programming simply means setting a time at which the device is switched on.

coffee

From a logical standpoint, a programmable coffeemaker is a clock radio.  Except it makes coffee.

Several other devices came up in the Amazon list with this same basic functionality.  There was an air conditioner that can be set to start at a given time and a fan that can be set to turn off after a particular delay.  But, the most common kind of product (after coffeemakers, of course) that supports this feature was “slow cooker”.  Similar to coffeemakers, which people like to be able to use in their sleep, slow cookers can be set to do their job while people are busy at work.

‘Stats

The slow cookers I looked at sported another class of programmability that I call “‘stats”.slow

In “probe mode”, the slow cooker can be made to turn off when a target temperature is reached.  Logically, this functionality is quite similar to that of a timer.  It’s about activating or deactivating a device, but it is based on temperature instead of time.  However, the reason that I consider ‘stats a different category from timers is that they are most often used in a feedback loop.  In particular, thermostats and humidistats (both of which show up in the Amazon list) and other kinds of ‘stats  measure some physical quantity (temperature or humidity, say) and hold it steady.  (The “stat” in “thermostat” shares its etymological history with the “stat” in “stationary”.)

What makes a ‘stat cool is that its sensor is sensitive to a condition that its effector corrects.  For example, a thermostat in a space heater triggers warming when the room gets too cold.  A humidistat in a dehumidifier triggers drying if the room becomes too wet.  Similarly, a barostat maintains pressure and a chemostat maintains chemical concentrations, although I didn’t run  across either of these on Amazon.  I remember many many years ago my father-in-law’s car had an automated windshield wiper that you might call a transparency-o-stat.  It activated the wiper whenever the windshield was obscured by rain.  It was pretty cool.

Whereas timers are most useful when a device needs to be turned on or off when you aren’t around, a ‘stat is most useful when there’s a quantity that can be controlled without interrupting you.  Perhaps one reason the transparency-o-stat didn’t become standard feature in new cars is that you are looking out the window anyway when you are driving—noticing that the window is wet doesn’t require any extra attention.

Schedules

Household thermostats are often bundled with another programming feature—weekly schedules.  In a sense, schedules combine timers and ‘stats with a “sensor” for the day of the week.  A schedule in a thermostat is useful because it’s common for people to try to save energy on heating or cooling during the day when they are at work or school.  But, during the weekend, things need to be handled differently as people are more likely to be home.  Schedule programming lets you specify target temperatures as a function of time of day and day of week.

The interfaces for programming schedules tend to be more complex because of the multiple dimensions (day of week, time, target) that have to be considered simultaneously.  The most extreme interface I ran across on Amazon was for a programmable towel warmer that lets you have a toasty warm towel waiting for you at 10pm on a Thursday, if that’s what you want.

towel

The complexity of this interface along with the seeming futility of its purpose (how many people know a week in advance that they need a warm towel at a specific time yet are unwilling to simply turn on a towel warmer as they get into the shower?) make me quite skeptical that this programmable device gets much use.  Of course, if we lived in a world where all our devices were programmable (hint, hint), it wouldn’t seem quite as odd.  Well, assuming you think towel warmers are normal.

The coolest example of schedule programming that I’ve read about is in the Nest thermostat, which combines schedules, timers, and ‘stats with proximity sensors (to find out whether you really are around on the weekend) and learning algorithms that adapt to your activity patterns and preferences.  I plan to write more about Nest in a future blog post.

Scripts

Amazon sells a dozen programmable breadmakers, and they embody a different kind of “programming”.  The process of making a loaf of bread consists of multiple steps like kneading, baking, waiting for the dough to rise, and, sometimes, adding in raisins.  Yum.  The timing of these steps depends on the kind and quantity of dough used as well as the preferred darkness of the bread at the end.  Programmable bread makers have scripts or sequences baked into their controllers (as it were) so you don’t have to spell out the steps yourself each time you use the device.

bread

The breadmaker is “programmable” in that you can select among a baker’s dozen or so of these pre-specified scripts.  The device is flexible enough to allow for a huge space of sequences, but the resulting complexity is mostly hidden from you since the useful sequences are encoded in advance.

Presets

Blenders, like breadmakers, can carry out sequences of actions with different timings.  For example, BarBoss Advance from Vitamix includes a set of scripts for producing different kinds of drinks.

blender

Each of these scripts is bound to its own digit on a dial on the face of the blender.  This means, apparently, that putting in the right ingredients, selecting Setting 4, then starting the blender will produce ice cream.  What’s not to love about programming?

blenderdial

The cool thing about this blender, though, is that you can connect it to a computer and create your own scripts.  You tell the blender how fast to go and for how long, how quickly to ramp up from one speed to the next, and when to add water or ice.

blenderprogram

The script you create is downloaded onto a chip that you can insert into the base of the blender, replacing the original pre-programmed scripts with your own “presets”.  Using a preset means that the complex sequence you’ve painstakingly created can be called back into being simply by selecting the right setting on the dial.

Presets are a great tool in the programming arsenal.  Other examples of devices with presets that I’ve seen include video gaming headsets/keyboards/controllers (allowing you to optimize your interface for “different gaming situations”, whatever that means), microwave ovens (allowing you to lay out different cooking stages that vary in time and power) and, of course, car radios (allowing you to jump to your favorite stations without having to hunt through the whole FM spectrum).

Composition

One of the delights of working with a true programming language is that it is possible to combine/compose programming features together in creative and powerful ways.  Although none of the devices I encountered on Amazon makes use of this idea, several do combine features in inflexible but interesting ways.  For example, the blender I mentioned uses scripts and presets together.  The same combination is used in message fans and fryers.  Thermostats and towel warmers use schedules and ‘stats.  The same combination is used in wall control / ventilation systems.  Rice cookers combine timers and presets.

All in all, I think it’s important to note which kinds of programmability have proven popular enough to be included in commercial products.  Whether and what kind of programming a product supports boils down to three factors:

  1. The behavior should be something that you can’t do yourself, either because it’s hard to remember (scripts, presets, schedules), boring or distracting (‘stats), or simply because you aren’t around to do it (timers).
  2. The behavior should be something that the machine can do on your behalf.  Noting the time or temperature is straightforward.  Remembering detailed sequences of mixing or baking steps is, you could say, a piece of cake.  Recognizing a face, understanding spoken dialogue, or moving with precision in natural environments is still quite harder.  (Although progress has been steady.)
  3. Finally, there must be a way of transferring your goals and intentions from your mind to the machine.  At present, the possibilities are still quite limited, but if devices supported general forms of programming and more people were familiar with this form of communication, the cost-benefit analysis would shift.

That pretty much covers it for the “programmable” appliances on Amazon.  As they exist today, it’s a little odd that they all are described using the same word.  But, if they supported a common programming language, it could make them easier to use and expose the deep connections between the different ways we can ask machines to work on our behalf.