My Connected Device is Listening

My Android phones have been listening to me for years. I have no doubt about this. When I started talking about something absurd late last year – “can you use hand lotion to condition a leather chair” – and then decided to look it up on my phone, the first suggestion google makes upon entering “Can you use” was “Can you use hand lotion to condition leather products”. And that’s just one example.

People call me paranoid for saying this, but I’m not. I just understand (or at least pretend I understand) what’s going on inside our connected devices.

Speech Recognition in the Cloud

If you have an Android phone with the Google+ Launcher applied, try this: Turn it on and just say “OK, Google”. This automatically opens the voice to text search box where you can talk to the device and get it to do things like a search or send an email or whatever. I’m sure the Fruit Company phone can do the same thing. And the one from Macrohard. It’s amazingly unamazing in a world where rapid technological advancement has made us jaded.

What’s actually happening here is really quite amazing: The phone is constantly listening for specific voice queues, and when they are triggered it starts doing stuff. It gets even more impressive when you start dictating. You can actually see the phone guessing and correcting itself in real time as it does to make sure it gets everything exactly right. And watching this happen it’s clear there is a lot of contextual semantic processing going on out there in the cloud.

What we have here is the dream of speech recognition come true in the cloud. And now that we have it people are (and should be?) terrified.

Your Samsung TV is Eavesdropping on Your Private Conversations

Earlier this week the tech media and everyone else suddenly got very interested in connected devices and their listening capabilities. In the Terms of Service for Samsung’s Smart TVs a line was discovered that said:

Please be aware that if your spoken words include personal or other sensitive information, that information will be among the data captured and transmitted to a third party

The reporting was quickly followed by a statement from Samsung saying:

If a consumer consents and uses the voice recognition feature, voice data is provided to a third party during a requested voice command search. At that time, the voice data is sent to a server, which searches for the requested content then returns the desired content to the TV

George Orwell fans with better memories than me immediately caught on to the striking similarities from the book 1984:

And the world burned for days.

The Clash of Dreams and Reality

What we have here is a clash of dreams and reality. In our dreams we want to be able to talk to our devices and have them do our bidding. In Star Trek they had the Universal Translator. You can now get much the same feature by downloading the Google Translate app on your smartphone. Try it. It’s absolutely mind blowing. But for this technology to work we can’t just rely on our phones or computers or TVs. Language is complex and can’t be simplified to algorithms that can run on our local devices. For this technology to work we need the Cloud. And that means literally recording and sending your conversations over the web to a server that then parses the data, gleans the meaning of it, and acts according to your instructions.

In short, when you talk to your device your device needs to actually understand what you are saying. Which is why your Samsung Smart TV and your cell phone and probably your computer and any other connected device in your house is in fact listening to you all the time

The Question Isn’t If You are Being Recorded, But Who Listens

Most people will find this revelation rather unsettling, but the reality is this is not new. It’s been going on for years. And it’s a direct response to what we as consumers have been asking for.

The bigger question is who’s listening. The device and service companies are pretty much unanimous in saying they are not recording and not listening. The recordings are purely for the computers. And in a way it is probably believable (unless you are talking about that website that wants to sell your face on a book. They are totally listening). I don’t actually fear the companies (much), but I do question their encryption algorithms.

Post-Snowden we have confirmation of what many of us have known all along: If you put it on the Internet the US intelligence system will be listening in. So does this mean that someone is sitting in a bunker somewhere in the US (or elsewhere) listening to our conversations while we watch TV, or eat dinner, or chat with our phones within reach? Not unlikely.

Somewhere George Orwell is shaking his head in shame while dictating his next novel through his phone.

The Spirit of Flickr and the Problem of Intent

I’m trying something new here: Audio versions of my essays. So, if you want to listen to me read this essay rather than read it, hit the play button below and let me know what you think about this idea! Over the weekend a conversation has started over the move by photo sharing site Flickr to […]

Simone 2.0

Simone 2.0: Custom backgrounds, sidebar position, RTL support, and more

After much ado Simone 2.0 is now live in the WordPress Theme Directory. The theme is cresting 50,000 downloads already and for the new version I decided to bake in some of the most requested new features. Custom Background One of the standard WordPress theme features that were not available in the original version of […]

Thinking Inside the Box – My WordCamp San Francisco 2014 Lightning Talk

What started as a challenge ended up as a lightning talk at WordCamp San Francisco 2014: Explain the CSS Box Model in an understandable way to a crowd of mainly bloggers and designers. In 5 minutes. With no live code. The video slides are unfortunately largely cut out of this video so you don’t get […]

Challenges with the new method for inheriting parent styles in WordPress child themes

The new recommended code for inheriting parent theme styles in WordPress child themes have uncovered several issues with the concept of child themes. What follows is a rundown of how things work, what problems are caused by the status quo, and finally some proposed solutions.

Why Android Fragmentation is Irrelevant to Web Designers

We’ve all seen them: Pictures of an array of hundreds of Android devices accompanied by some snide remark about how “Android Fragmentation” is making it impossible to build and test anything for the mobile web. I’m here to tell you that’s not true. You content doesn’t care about the size of the visiting browser. Neither should […]