Good Morning Karen. Cool or Scary?

Last month I spoke at a telecommunications industry event. The briefer before me showed a video by the Hypervoice Consortium, titled Introducing Human Technology: Communications 2025. It consists of a voiceover by a 2025-era Siri-like assistant, speaking to her owner, "Karen." The assistant describes what's happening with Karen's household. 15 seconds into the video, the assistant says:

The report is due today. I've cleared your schedule so you can focus. Any attempt to override me will be politely rebuffed.

I was already feeling uncomfortable with the scenario, but that is the point at which I really started to squirm. I'll leave it to you to watch the rest of the video and report how you feel about it.

My general conclusion was that I'm wary of putting so much trust in a platform that is likely to be targeted by intruders, such that they can manipulate so many aspects of a person's life. What do you think?

By the way, the briefer before me noted that every vision of the future appears to involve solving the "low on milk problem."

Comments

Ron Trunk said…
How does that movie line go again? Oh I remember: "I'm sorry, Dave. I'm afraid I can't do that."
Unknown said…
I see my children taking advantage of that. I'll be the old geezer saying that contraption away from me.
Anonymous said…
Everything crucial needs a manual override. That's like one of the fundamental principles of sci-fi paranoia.
dre said…
Yes, in Sheridan and Verplank's Ten Levels of Automation of Decision and Action Selection -- http://images.slideplayer.com/2/730405/slides/slide_8.jpg -- we would typically want automation to be at best level 6 and at worst level 4. Karen was working at level 7 automation, which I agree is too much -- especially given the context of her situation.
Anton Chuvakin said…
Well, we actually recently started a mini-research project on that very topic: http://blogs.gartner.com/anton-chuvakin/2015/07/10/on-evil-ais-and-evil-people/

In scope are AI/smart machine/robot exploitation...
dre said…
Anton: wow that work very much correlates to my provided references!
Unknown said…
This is interesting - but I think many things, even physical barriers like bi-folding gates, can be subject to attacks in this day and age.
Anton Chuvakin said…
@dre Thanks --- we are hard at work on this [despite some people making fun of us for looking too far in the future]. I will blog some notes from our work fairly soon
Anton Chuvakin said…
@anonymous Manual override, BTW, is a non-starter for A LOT of "AI"/smart machine uses -- trading, even driving [like you are watching a movie in your car and the driving robot say "yo..now you do it; it is too hard for me!" FAIL]
Anonymous said…
Personal information online is targeted by hackers all of the time...but this is REALLY personal information...you're just waking up, the coffee is brewing in the kitchen, your mother 's ETA is 5 minutes, it seems to know where everybody is at any given time...including your children! This is a cyber-predator's dream.
That's information that just shouldn't be available, let alone 24/7.
If you can GUARANTEE that nobody else can access the data, I'm sure that there are more than a few governments and financial institutions that will pay dearly for that technology.

Popular posts from this blog

Zeek in Action Videos

New Book! The Best of TaoSecurity Blog, Volume 4

MITRE ATT&CK Tactics Are Not Tactics