Abstract
Psychology allows humans to understand others at a fundamental level and assists in understanding why people take specific actions in certain situations. It is a social science that can assist with mental health issues, business practices, and education, among many other applications. Social psychology (SP) focuses on how behaviors and actions can be analyzed in a greater context within organism interactions. With the innovation of programming and eventual machine learning (ML), we decided to investigate how computer science (CS) has affected research within nonverbal communication (NC), a subtopic of SP. Algorithms within the fields of classification and computer vision are used numerous times, as they are extremely useful in the analysis of behavior and facial expressions, two crucial topics which form the foundation of NC. Through our research, we found that explicit interdisciplinary work between NC and CS was scarce, but there have been studies coming close to achieving a higher understanding of their relationship. Thus, we will report on research done in the foundations of NC and discuss how CS may assist in progressing the field.
Introduction
SP is a discipline that, similar to biology and physics, is studied and progressed through empirical research. It is a social science dedicated to the study of how people think about, influence, and relate to each other. Therefore, SP is extremely useful as it can be applied to many areas with wide social importance, such as aggression, tyranny, and prejudice. It is also generally useful for many other areas of life and politics such as crime, taxation, and health, as it helps to understand the social causes of those specific domains1. Though we will not consider these specific aspects in our review, it is important to consider the various applications of SP.
Nonverbal communication (NC) is a subfield of SP and can be defined as any type of communication that goes unspoken (e.g., gestures, facial expressions). NC exists in everyday interactions and is a growing branch within SP. Though much research has been performed on NC, there have been little to no comprehensive reviews of the interdisciplinary work relating to computer science (CS) and NC. This paper will therefore attempt to fill in that gap by analyzing first recent research on NC and afterwards considering methods by which the field may be moved forward with CS.
Structure
We will now analyze the background of SP, starting with philosophy in the 18th century which evolved into numerous social theories within the 20th century, building the essential foundation for the discipline. This will provide adequate context for the research we analyze later regarding NC. Specifically, we will discuss gestures, dating, and fairness through three different case studies. Then, we will focus on how CS has assisted NC through machine learning (ML) algorithms. Finally, a discussion of the interdisciplinary potential of these fields will be provided.
Background on Social Psychology
Philosophy
The foundation of SP has its beginnings traced back over two centuries ago to the works of scholars David Hume and Immanuel Kant and was heavily reliant upon the philosophy of the 18th century. David Hume’s teachings in his famous work A Treatise on Human Nature strived to form a psychological basis for human nature, and he argued that passions governed human actions instead of reason2. He additionally dismissed standard notions of cause-effect relations, arguing that they were based on grounded habits of thinking rather than causal forces in the external world. His arguments can be seen as an initial foundation for many topics that would be later considered parts of SP, such as attitude and the self.
Immanuel Kant, on the other hand, criticized the way researchers handled psychology at the time and argued that it should not become a type of empirical science like the natural sciences or mathematics3. He argued that empirical sciences were possible because they operated on things that have a spatial dimension, but psychology inherently relied on a subjective consciousness that had no physical extension in space. Ironically, Kant’s criticisms were shown to help develop psychology as an empirical discipline as other philosophers (namely Johann Herbart and Friedrich Beneke) attempted to combine psychology with mathematics in order to present it as a scientific discipline.
The early history of psychology helps us to understand how the discussion of more specific social topics like NC came into the lens of research. These philosophical ideas are the backbone of modern psychology and help us to further understand the discipline as we now go into our case studies.
Case Studies of NC research
Gestures
NC within the learning environment has been researched thoroughly. An important notion learned is that teachers’ gestures can be extremely influential towards a student’s understanding of major topics. They can either help or harm a student’s understanding depending on the way they are used. It has been shown in previous research that teachers have used gestures which unintentionally harm a student’s understanding, such as in a certain mathematics problem where the teacher used a hand gesture that suggested an incorrect arithmetic operation4. However, within that same study, it was shown that gestures were helpful in instruction when they conveyed a correct strategy for solving a math problem that was different from one stated in the accompanying speech. This has many implications for schools as it is not commonplace to train teachers on NC methods, so they may inadvertently influence a student’s understanding of mathematics without realizing it.
Dating
Another question that can be raised regarding NC is the extent to which it assists in an individual’s dating life. One study by Carroll and Gilroy found that homosexual people were much more likely to recognize other homosexual individuals based on their nonverbal characteristics and brief interactions than heterosexual people5. This was done using a 2×3 multivariate analysis, with a personal assessment of perceptual accuracy and the sexual orientation being the independent variables. This research thus proposes the idea that people of a certain sexual orientation are inclined to recognize others of that same orientation. However, appearance was also a major factor in considering the orientation of the participants, alongside NC, and the question of whether heterosexuals could identify other heterosexuals in this manner was not answered by the study. Thus, it can be reasonably concluded that these nonverbal behaviors help homosexual individuals, but not necessarily heterosexuals in the same way. Another study by Frith details the importance of facial expressions in social interactions by stating that they can be deceptive if the person signaling the emotion does not actually feel it, which has obvious implications within dating6. Taken together, these studies are limited but display crucial nonverbal components within the complex behaviors associated with dating.
Fairness
Fairness is the next subtopic of NC that we will discuss. How do people react when something unfair in their eyes has been done to them? One study by McCall and Singer shows that fairness violations elicit strong avoidance-related responses by victims, as they feel a variety of emotions such as disgust, anger, and sadness7. The study does this by putting participants in an economic game where some players play unfairly, thus making it possible to see how the others would respond. The fair players played the “Unfair Others” (as coined in the study) by monetarily punishing them and turning their backs on them in the virtual environment. Proxemic imaging, a unique type of imaging introduced by McCall and Singer which uses frequency maps of digital tracking data, allowed participants to do this and to keep fair players close to themselves.
However, humans are not the only creatures who demonstrate an aversion to unequal equity. A study by Brosnan & de Waal in 2003 demonstrated monkey aversion to an unequal food award when performing the same amount of effort as another monkey8. Monkeys would exchange “tokens” with human researchers for specific food items, namely cucumbers or grapes (grapes being the most favored award). The study analyzed exchange behaviors and displayed that the strongest increase in refusals to participate in the exchange occurred when a partner received a more favorable award for the same amount of effort. This, as illustrated by the study, supports the theory of an early evolutionary origin of inequity aversion.
Applying Computer Science to Nonverbal Communication
Now that research of NC on its own has been looked at, applications of CS to NC must be considered in order to view how CS has helped progress the knowledge of nonverbal behaviors. Specifically, a branch of CS which is broadly defined as the capability of a machine to imitate intelligent human behavior. It focuses on building methods/algorithms that leverage data to improve performance on some set of tasks.
Mental disabilities can provide a crucial lens into the field of NC. People with conditions in the autism spectrum and those with severe communication issues in general, face issues such as social phobia, avoidance behaviors, and self-injurious behaviors, among many others9. It has only been quite recently that extensive research has been done into the field of finding ways to assist people with mental health disabilities. One study attempts to classify individuals with autism spectrum disorder (ASD) by using ML. The research methods utilized a classification algorithm that was trained on intrapersonal synchrony from 29 autistic and 29 typically-developed (TD) individuals. The classification algorithm was a success, only misclassifying one participant who had ASD10. This provides a fantastic way for detecting underlying mechanisms of autism and thus further research on the subject is encouraged.
As described above, facial expressions play a huge role in social interactions. With ML, many algorithms have been created to analyze facial expressions and thus, emotions. One study by Metaxas and Zhang reviews many facial recognition algorithms and motion analysis is also used to analyze nonverbal behaviors. A drawback that was considered in this study was that the motion analysis methods were largely based on traditional 2D models11. This was not optimal as it was very difficult for 2D models to handle large off-plane pose changes, such as rotations or head tilts. Solutions are offered, such as training multiple 2D models at different rotations and then switching between multiple models during motion analysis.
Discussion
SP explores the human mind and attempts to find out why humans partake in certain actions and what the deeper meaning behind those actions is. As seen from the various case studies on NC, there has been much progress made on the topic. Gesture, dating, fairness, and inequity aversion have been covered thoroughly from a non-CS standpoint. On the other hand, the studies with CS employed algorithms within the fields of classification and computer vision, as they are extremely useful in the analysis of behavior and facial expressions. They displayed promising results for the classification of ASD and analysis of facial expressions. However, these studies offer a limited understanding of the influence ML could have within the field. Though we are not suggesting that no studies are attempting to go into these topics, it is an interdisciplinary field mostly incomplete, with the studies shown being the current pioneers.
For future research, the field of NC could be furthered by applying these ML models more explicitly to topics like gestures and dating. Even topics only discussed within the intro of this review, such as aggression, prejudice, tyranny, and crime, could be understood deeper through ML classification algorithms. Other examples of ways to continue research into this combined field include applying classification algorithms to the learning environment and to fairness. There could also be a model which capitalizes on the data found by Carroll and Gilroy which classifies specific behaviors associated with heterosexuality and homosexuality. Overall, researchers have scratched the surface of what can be done with ML when it is applied to NC, and this field serves as a way to invoke thought about what can be learned from the study of the human psyche. Our broad analysis of these papers serves as a starting point for future researchers to combine ML with uncharted territory in NC and should be treated as such.
- Sutton, R., & Douglas, K. (2019). Social psychology. Bloomsbury Publishing. [↩]
- Fieser, J. (n.d.). David Hume (1711-1776). Internet encyclopedia of philosophy. Retrieved November 9, 2022, from https://iep.utm.edu/hume/ [↩]
- Teymoori, A., & Trappes, R. (2017, June 8). A revolution in thinking. The British Psychological Society. Retrieved November 9, 2022, from https://www.bps.org.uk/psychologist/revolution-thinking [↩]
- Goldin-Meadow, S. (2014). How gesture works to change our minds. Trends in neuroscience and education, 3(1), 4-6 [↩]
- Carroll, L., & Gilroy, P. J. (2002). Role of appearance and nonverbal behaviors in the perception of sexual orientation among lesbians and gay men. Psychological Reports, 91(1), 115-122 [↩]
- Frith, C. (2009). Role of facial expressions in social interactions. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3453-3458 [↩]
- McCall, C., & Singer, T. (2015). Facing off with unfair others: Introducing proxemic imaging as an implicit measure of approach and avoidance during social interaction. PloS one, 10(2), e0117532 [↩]
- Brosnan, S. F., & de Waal, F. B. (2003). Monkeys reject unequal pay. Nature, 425(6955), 297–299. https://doi.org/10.1038/nature01963 [↩]
- Autism Tasmania. (2022). Common Challenges. https://www.autismtas.org.au/about-autism/common-challenges/ [↩]
- Georgescu, A. L., Koehler, J. C., Weiske, J., Vogeley, K., Koutsouleris, N., & Falter-Wagner, C. (2019). Machine learning to study social interaction difficulties in ASD. Frontiers in Robotics and AI, 6, 132. [↩]
- Metaxas, D., & Zhang, S. (2013). A review of motion analysis methods for human nonverbal communication computing. Image and Vision Computing, 31(6-7), 421-433. [↩]
