There’s reason to worry about AI reading emotions

One application that comes to mind is a toy robot called Moxie that incorporates multimodal AI emotion recognition in its engagement with children. Based on a paper released by its creators, the behavioral metrics that the toy tracks primarily relate to facial expressions and word choices.

Here, even though the word choices are technically recorded via speech through a microphone, it’s different from SER because the analysis of words is presumably powered first by a speech-to-text model that converts the speech into text, and then analyzes that text to examine if certain words, such as “family” or “friend,” relate to concepts that they deem to be “positive” or “negative.”

This is generally called “sentiment analysis” in the field, and it’s also a somewhat contentious area for similar reasons: words alone are not consistently indicative of “sentiment.” The paper states that the toy was first developed as a tool for supporting children diagnosed with mental behavioral development disorders or MBDDs, but my understanding is that it’s now being sold as a more general learning companion for all children that supports “holistic skill development,” which of course expands the addressable market of Moxie.

My colleague Mara Mills has called this phenomenon of resourcing disability as a step towards more profitable realms as “assistive pretext.” As I briefly recount in my paper, children, and especially those who have been diagnosed with MBDDs, have historically been designated as the target demographic and justification for the initial development of emotion recognition technologies.

A chapter in Rosalind Picard’s pioneering 1995 book Affective Computing, for instance, has a section dedicated to “helping autistic individuals.” About a decade later, researchers from the University of Cambridge also proposed an “emotional hearing aid” that was described as a facial prosthetic to help children with Asperger’s syndrome socialize.

To my knowledge, most of this work as it has been taken up by the broader tech industry has now developed beyond these “assistive pretexts,” and the benefit for the individuals that served as the justification for their initial development is contestable. My hope is for researchers and builders to remain critical and compassionate in their development, or not, of these technologies.