This was one of four sessions at the Microsoft Social Computing Symposium 2015 that took place in San Francisco over three days. Liveblogging by Catherine D'Ignazio and Alexis Hope.
Session organized by Kati London - @picklesnumber1
What are the implications when our objects can talk to us and to each other? What are the possibilities and challenges? This session addresses the consequences of objects and systems that speak or act like humans and how we engage with them socially and emotionally.
Yamove! partnered dance game by Katherine Isbister and colleagues. Photo from the Yamove! website.
Playful Relations with MachinesKatherine Isbister - @kcisbister
She is housed in Brooklyn in a school of engineering. Focused on CS side of game development but looks to partner with other places like the NYU Game Center. Prior to working with games, she was working with HyperCard and Cliff Nass. He and his colleague wrote a book called The Media Equation. People would treat tech as if it were human if the computer acted like it was human. If a computer did a favor for you, then after you went thru training and the computer asked for a favor you were much more likely to help it. Her thesis was about body language. You had to do a desert survival task - which things do you rank the highest in a survival situation with the computer advising you.
Then the Microsoft Word released the Paper Clip. Her advisor worked on the project. It was supposed to be a helpful agent but everyone hated it which made Isbister sad about the future of emotional computing. She shows a screenshot from a rhythmic rapping game from the 1990's. You have to do these tasks by playing a rhythm game. She was charmed by the game. Video games have characters and everyone loves them. Anthropomorphizing in this domain is acceptable. Games have always had little creatures in them - even early games like Centipede. She taught classes about game character design. She wrote a book called Better Game Characters By Design. The best game characters were using cues from human social life.
Context Matters
The failure of Clippy the paper clip came at a time when computers were primarily work machines. People expected it to be quiet, focused space. They were not used to socialization from the computer. Whereas firing up a video game, you understood it would be social and interesting. Part of it was context. Also, the desktop metaphor gives you very little info about context of what they are doing. Whereas a video game has context, narrative and much more control over the world that the person is in. Clippy was kind of mansplaining things you already knew, e.g. "It looks like you are trying to write a letter…"
When she was working on this project, she interviewed people that played games about their emotional engagement. One of the games that came up again and again called Planetfall. It was a text-based game. Floyd was a robot that helped you. At a certain point he sacrifices himself for you. This made people burst into tears. He had done something genuinely meaningful for you.
Flashing forward to now…
She is working on a book for MIT Press on a book for the general public to understand what games are doing to affect people emotionally. Forging a relationship over time is an important part of this. For example - Japanese Dating Sims - you spend days and days in the game. There are 3-4 people you try to pick to court. You spend a hundred days in the game carrying their books, befriending their brother. Maybe a character will decide to go out with you. It's a sweet interaction to have with a fictional character.
A guy in Guam staged a wedding to his character. People enjoy these relationships. It feels better to play a dating SIM for some people who say they enjoy being able to follow the rules and know that they will have a successful relationship outcome. She references the movie "Her" by Spike Jonze where the main protagonist falls in love with his AI assistant.
She created a game in Japan that had a character intermediate a conversation between two people. One version offered safe topics and another version offered more provocative topics (e.g. whale killing). People loved scapegoating and blaming the character for offering awkward suggestions and ended up bonding with each other over it.
In the last couple years she has been building games for physical spaces. Yamove! is a dance game that uses mobile phones. Movement builds connection, for example dances at weddings. They took BBoy dance battles as a metaphor. They took mobile phones and strapped them to people's wrists. You and your partner face off against another pair of people. The pair that is most in sync with each other wins. The whole point is to build connection between the dancers.
The final project is a collaboration with an artist who was interested in how people take on different characters' personas in games and wanted to see if you could translate that to the real world. She created wearables that you could put on. Person with backpack collects energy from others and then there's a relationship with a gauntlet.
She has moved to making technologies that artfully humanize relations between people.
Tom Coates - @tomcoates
How do we want to communicate with a world of connected objects?
What about objects with personalities and expression in particular. Awhile back he founded Product Club with a friend of his, Matt Biddulph. His other projects weren't successful but were critically celebrated. They started a year of interesting projects. Matt made a mobile phone. Another was QueryApp which was a Q&A app. It wasn't sexy enough to be their "Taylor Swift". They did some work on a coffee app.
One of the areas they really cared about was the Internet of Things. One of the experiments they did was a project called House of Coates. He shows a video of his house. The idea was for him to live within an Internet connected house, so he went out and bought all of the products he could find in this space. He connected numerous objects in his house - thermostat, motion, and so on. Then he made a Twitter account for his house, which narrativizes his house. It shows which lights are on, what is going on. It is carefully scripted to cover the gaps that he couldn’t determine with current technology. So lots of it is “smoke and mirrors.” At any given time, only about half of it is working.
The weirdest part of this was trying to figure out what the “feel” of the house should be. What should the tone be? How did he want his house to engage with him? But he didn't want it to be shouty or loud. He wanted a calm and reasonable tone of voice that would work in most circumstances.
He liked the house talking to him after awhile. He wasn't the only one to do this. Members of the public would also chat with his house. It was a small project of a small section of a year's work. It's an illustration of what you could do if you connected your home.
Then an article was published in MIT Technology Review. In the article, the journalist really seemed to get what he was going for, and wrote it up in a pleasant and appealing way, according to Tom. That spawned other articles, one in the London Metro that goes out to people riding the London tube. This piece also was fairly aligned with what Tom was going for, and he was surprised that the angle was not ridiculing him. After that, came dozens of other articles. Lots of people wanted to interview him, and were surprised that the house itself looked really normal and “most of the magic was happening on the cloud.” Then he ended up in Wired magazine, NYT Real Estate section and Chinese TV.
He shows a video from CNBC with an interview of him and videos of him in his house. The angle for this piece was along the lines of: “What if someone was watching you all the time...watching you to try to make your life easier?” He says the video made him extremely uncomfortable.
The idea of an anthropomorphic home, a home with a voice, really resonated with people. Almost no one said "why are you doing this?" Almost everyone said, "I wish my home could do that." Why voice instead of buttons and touch screens?
A couple people like Adam Greenfield, Kati London, and Yuri have done work on bringing social object theory into the digital world. Matt Jones - author of the Demon-Haunted World - has also done work in this space. There are two concepts in play:
- Animism - we have a tendency to anthropomorphize things. We attribute souls to plants, inanimate objects and natural phenomena.
- Inscrutability - we anthropomorphize things more when we don't understand them
For him the interesting point is when we transition from not anthropomorphizing to anthropomorphizing. He thinks this is when they are inscrutable to us. When they are doing things we don't understand we resort to human-to-human dialogue. There is not another area that is more inscrutable than the internet of things. There is this sense with that of objects operating with intent. For example, he was at home and the lights came on suddenly. He still doesn't know why. When objects are talking with one another, they react to one another, optimizing themselves to make your life better but puzzling you.
If these objects could tell you why they were doing things in a more human way then that would be more comforting. That lends itself to human-like qualities. The objects shout. They react to things. They want to talk to you on their schedules. The desire for the object to communicate with you. He references a talk by Matt Jones and himself at Web2.0 Expo in 2008 that started discussing the politeness of objects.
When you start talking about politeness with objects, you’re veering into the territory of anthropomorphizing. He describes a pet peeve for beeping washing machines - “how rude is that?” He describes a scenario of taking a nap, the machine beeping, and you not wanting to go tend to the needs of your wailing washing machine. It is “communicating on its own channel and being rude.” He tells the story of the robotic vacuum cleaner that just sits there and yells in "mum voice" when it gets stuck. It would be so easy to imbue these objects with less, more or different personality.
It would be better if these things talked to us in channels that we understand. We want them to be polite and discreet. Don't expect us to run to the phone and deal with it right then.
He shows a project called “Courtney Love’s Medicine Cabinet.” It tracks how many pills have been taken out of it, and if it crossed a threshold, it tells an IRC channel “don’t listen to her, she’s high.”
The point is that we can use anthropomorphism to make objects comprehensible. We can make them polite and discreet. We can also form a stronger relationship with the objects in your life if you can relate to them in these ways and with emotional bonds.
Adam Greenfield theorizes the dystopian view and risks of this space.
He ends with a few potential consequences. One risk is objects that are absurdly cute (he shows a picture from the movie The Brave Little Toaster). Do we want objects to be chirpy and whimsical and infantilizing? Couldn't that get annoying?
He shows a video of "The Nozzle". Another risk is objects that are brutalist and unfeeling.
And a final one would be the singing teapots and kitchen of Beauty and the Beast which nobody wants.
He briefly mentions that he is working on Thington, an attempt to create connected objects that avoid some of these potential consequences.
Darius Kazemi's Yearly Awards Twitter Bot
Darius Kazemi - @tinysubversions
Design Considerations for Bots and Autonomous Agents
Kazemi's best known project is something called Random Shopper. What happens when you order something from Amazon and it doesn't arrive for two years? Or something that buys what's in your recommendations list? Random Shopper searches Amazon for random words and then buys stuff for under $50 that meets that criteria. It's a bot but doesn't live on social networks, manifests through the USPS.
He shows a Tumblr bot called Reverse OCR that grabs a random word and scribbles until an OCR library recognizes it as the word. People on Twitter didn't like it but on Tumblr people went crazy for it.
He talks about how context is important. People love to ask him AI questions, which he has done in the past, but he is not doing that kind of work with these bots - it is all smoke and mirrors (which, he says, is how AI in video games works too). He shows how he transformed a dictionary into a joke generator by mashing it up with the “Girl, you must be…” meme. It's computationally very simple by using contextual cues.
The patterns he uses are that he looks at repetitive human behavior in the wild. For example, the labor of fans on Tumblr. People like to make anigifs of their shows and put them on Tumblr. His bot goes through the Wire and extracts text and image and puts it on Tumblr. It does thousands an hour. He likes to see them go out "into the wild" and people use them.
Another format he copies is the practice of turning song lyrics into charts. He generates charts by sourcing questions and answers from Twitter.
He describes a technique called “weaponized humor” where he copies a joke format and uses a bot to copy it and spew it out on twitter. His goal is to ruin that joke format because he thinks it’s terrible, for example AmiRite Bot.
Another bot is TwoHeadlines which mashes together two headlines from trending news articles. He gets a lot of people who use these as reference points. TwoHeadlines is a simple algorithm that scrapes Google News from a given category, clicks on a topic at random, and grabs the headline. It knows what the subject of the headline is, and replaces that subject with one from another headline.
A more recent project is YearlyAwards. You follow it and it gives you an award.
Content, Forever is a project inspired by looking at Medium Looking at “long, detailed, meaty think-pieces on Medium” that get deep into a topic but from them you learn “nothing about the world.” In this bot, you put in how many minutes of content you want, and then a topic you’d like to read about. What this actually does is “automate the process of falling down a wikipedia hole.” It goes to the wikipedia page and grabs links from the first 5 paragraphs, pulling content and and tracing the links. He copied the CSS and format of Medium. His goal is to automate and make rote the things people do online.
He organizes BotSummit every year where people who make bots get together and talk. He was happy to hear Tom talk about rudeness in the prior talk. He tries to design things with the people who are using them in mind. For example, TwoHeadlines pulls jokes from massacres. He had to make a call in that case. He makes it clear that this thing is a bot. One of the greatest tricks he pulls to make them less offensive is to label them as bots. He does have an extensive blacklist of terms that his bots will never use.
He describes the curation he does with his bots. For example, one joke TwoHeadlines sometimes tells involves replacing the individual in a sentence with one who has a different gender, leading to transphobic jokes. He doesn’t like this, so he is actually working on a transphobic joke detector so that his bots won’t make these kinds of jokes.
Julian Bleecker - @darthjulian
The TBD Catalog
Bleecker works in a design studio called Near Future Laboratory. He is also an engineer. He did a Kickstarter book recently.
Different archetypes for how people represent what might be possible. For example, on Kickstarter people look at the page and say "Oh I want one of those". He loves that as a means to open up conversation about crowdfunding or other aspects.
Tangible Representation - About what it might be like to inhabit the world we are building. A world of 3d printers in every room, ear fresheners, and so on. In many cases, they are disappointments in all kinds of normal ways of not working and so on. For example, they did a newspaper that was a supplement that was about "What would sports look like in the future?" They said it'd be sports + data, and extrapolated in various ways about what it would be like. They speculated that a lot of tech moguls would end up buying sports teams for the data and that is actually happening now.
He did another project called “Curious Rituals.” This was a project where they looked at how people might interact with objects in the future. The car uses facial recognition to start the car, but the character in the video was wearing makeup when she configured it, so every time she wants to start the car she has to make up her face like she did when she first configured it. This project is meant to call out designers and developers for not finishing their work, creating systems that don’t really work in real life.
He shows a slide about "Wouldn't it be cool if…" to explore possible futures like drones everywhere, bathroom door APIs, aquariums on the Internet, and every thing connected to every other thing. He doesn't feel like we go far enough to explore the actual implications of these possible futures. Like, does everyone have a new trash bin just for their 3D printed trash? His work tries to explore these futures and tell their stories. It makes the design more robust and more fun.
When people talk about things like the Internet of Things at events like CES, he asks: “Can you be more specific about that world? What do you mean?”
It's a design challenge. Can you be more precise? What are the consequences and implications of that world? He wants to play out these questions and worlds and tell rich stories around them.
He gathered a group of people in 2012 to create a product catalog for the future called the TBD catalog. Take the current memes and extrapolate them into the future. He used the form of the product catalog - the SkyMall of the future.
They spent a day in the archives of the Henry Ford museum where they have the entire collection of McDonald's Happy Meals. They documented their process extensively and produced a catalog of design fictions. He describes how they didn’t want to make the catalog about predictions, but about provocations. It was kind of like flipping through AliBaba but on LSD.
Fox News called them to talk about their weather technology. His semi-serious goal is that design fiction should be employed routinely. Better than a consumer panel. It is a kind of design research method to bring creative insight and storytelling to enliven things. Another archetype they are doing is an industry report on the Internet of Things. The industry report will have pundits - recurring characters that help you navigate this new world by providing product advice and guidance about emerging trends. For IXDA this year he’ll be doing a workshop on creating “QuickStart” guides - one page instructions for how to use hypothetical design fictions.
