Introduction

Everyday judgments made by untrained laypeople possess some degree of accuracy in predicting a range of psychological characteristics, including personality, intelligence, political ideology, and sexual orientation. Deep learning approaches have provided significant predictive improvements over layperson judgments in these and other areas. The success of deep-learning approaches for predicting psychological characteristics is not only socially important, given privacy protection concerns, but scientifically intriguing as well. To provide deeper psychological insights and to help the public identify ways to protect sensitive information, the use of deep learning approaches should not stop with successful classification. We should also explore what it is about a face that communicates the sensitive information. Are deep learning approaches simply confirming and independently detecting previously-posited facial indicators of a given characteristic, or are they picking up on salient facial differences which were previously unrecognized?

This article explores the relationship between faces and ideology using a large, publicly-available sample containing both facial photographs and ideological data-namely, 3233 Danish political candidates for local office. It uses a wide range of techniques, including convolutional neural networks, heat maps, analyses of facial expressions, and assessments of physical characteristics such as masculinity and beauty. By integrating these approaches, the article goes beyond identifying the degree to which faces connect to ideology in two ways: First, it identifies specific features of the face that connect to the model’s predictions for political ideology.

Faces and Ideology

The study of faces in politics has a long history. Early work found that voters perceive their preferred candidates to be more attractive than opposing candidates. Voter perceptions are themselves politically consequential: For example, faces evaluated as more competent were substantially more successful in an election, and attractiveness provides electoral bonuses as well. This literature has generally posited the relevance of faces to politics to be limited to subjective impressions of those faces. A recent study strongly refocused the argument by demonstrating that algorithmic classification of faces led to substantial accuracy when predicting ideology. This study’s focus, however, was to demonstrate the significant privacy threat posed by the intersection of deep learning techniques and readily-available photographs. Such investigations are in part needed simply to confirm the relevance of faces per se.

No study has yet employed heat-mapping to identify which segments of facial photographs are informing the classifications provided by the deep learning algorithms when it comes to faces being classified as being rightist or leftist. Because standard cropping procedures for facial photographs (including those employed in prior deep learning facial studies) leave non-facial information in the image, heat-mapping is important to discern whether information beyond the face is informing the classification of faces.

The use of heat-mapping extends beyond demonstrating potential needs for methodological refinements, however, in also highlighting which facial features are the most informative. This facilitates a second area of that needs exploration: Why are faces informative for ideology? That is, what information is a neural network detecting when it learns how to correctly identify which faces belong to those endorsing a given ideology? The social importance of the question is clear from a privacy protection perspective: Facial photographs are commonly available to potential employers, and those involved in hiring decisions self-declare a willingness to discriminate based on ideology. Members of the public may thus be aided by recognizing what elements of their photographs could affect their chances of employment. But understanding the facial elements responsible for successful predictions of associations between faces and ideology is also important for the study of ideology. Knowing that faces detectably differ based on ideology is undeniably important but identifying the specific features of interest may help to illuminate the origins of ideological differences.

Read also: Get Rid of Facial Hair

Several plausible candidates exist. Some of these concern facial morphology, i.e., stable physical characteristics of the face. For example, politicians on the right have been found to be more attractive than those on the left. Higher right-wing attractiveness scores were also observed among young (male) adults in the general population, though not among pundits. Masculine facial features may also be relevant: Masculine characteristics such as upper body strength appear to be associated with conservative economic preferences, and a preference among conservative voters for masculine-looking males is well established.

Facial expressions may also be relevant. A recent study reported that computer-coding of facial expressions were associated with self-reported ideology, with those on the left showing more surprise and less disgust. This study focused on quantifying the degree of threat to privacy, reported only the degree to which other facial expressions could independently facilitate the classification of ideology, and not the direction of any other associations.

However, the observation of an actual association between any given expression or aspect of facial morphology does not entail that this association is being detected by the algorithmic processes used to classify the faces by ideology. No study has yet tested whether model-implied ideology correlates with any facial characteristic, whether morphological or expressive. As such, the degree to which these features are informing deep-learning approaches and thus represent a privacy threat needs further consideration.

The Present Study

This study begins by exploring how computational neural networks classify political ideology from a single facial photograph. It then uses heat maps to examine how non-facial information contributes to these classifications and to identify facial features of interest for future study. It also examines how a range of characteristics of faces-masculinity, attractiveness, and expressions-connect to model-predicted ideology. The goal is not maximizing accuracy per se-had the researchers explored every model architecture and every set of hyperparameters, it is likely that the accuracy presented here would be increased. But the question of whether facial photographs can predict whether someone is leftist or rightist is well answered in the affirmative. This study instead explores why those predictions are possible, contributing to “interpretable AI” by using theory-guided facial features to seek to identify why deep learning can successfully classify faces by ideology. Thereby it moves beyond the many such studies which seek to peek within the “black box” of deep learning by using features of the model itself. The present work is thus part of the social data science tradition, in which machine learning methods are combined with theories and empirical results from traditional social science.

Methods

Data and code availability statement

The data has been processed in accordance with Danish and European GDPR regulations according to the University of Aarhus Data Controller Unit. Scripts to redo the analyses will be made available on OSF upon publication. The study was not preregistered. The informed consent of participation and publication of the facial images presented in this article was obtained from the participants.

Sample: Municipal candidates

The primary dataset consists of 5230 facial photographs of political candidates from the 2017 Danish Municipal election. These photos are a publicly-available resource, provided for use in the public sphere by the candidates themselves to the Danish Broadcasting Cooperation (DR). Danish municipal elections take place in a non-polarized setting. Candidates running for office in these elections have not been highly selected through competitive elections in party organizations or by participation in high-stakes elections, and they are thus described as the “last amateurs in politics” by Danish political scientists.

Each candidate’s ideology (dichotomously scored as left- or right-wing) was assigned based on the party label under which the candidates ran-see the list of specific parties in Supplementary Materials 1. Members of local parties with less-defined ideologies could not be readily assigned an ideological score and so were excluded prior to any analyses, leaving 4647 candidates, of which 1442 were female. These images were then manually inspected by an author blind to the candidate’s name or party. Additional exclusion criteria applied at this time included: (1) the candidate’s face was either not located during pre-processing, not shown in the photo, or not provided in sufficient resolution; (2) the photo was not in color (which would prevent pre-processing all images via gray-scaling in the same way); (3) the candidate was the candidates who did not appear to our rater to be of European ethnic origin, as this small subset (118 individuals) skewed very highly towards left-wing parties, with 2.5 × more representation there than among right-wing parties.

After applying these exclusion criteria we had 3288 candidate photos remaining-this is our full primary sample. We then separated the males and females before using R to sample each gender into training, validation and test samples with probabilities of 0.7, 0.15 and 0.15. Ns for each sample are provided in Table S1.

Because beards can impair some of the subsequent analyses (such as the detection of facial expressions), we produced a “reduced” version of the male training, validation, and test samples in which those with beards (N = 657) were dropped. This sample is used for our analyses that involve “identifying salient characteristics”.

Replication sample: Danish parliamentarians

To provide a second test for the accuracy of our algorithmic predictions of ideology based on faces, we obtained an additional sample, namely Danish Parliamentarians. The sample was created following the procedure described for the primary sample, omitting the final two steps. That is, other than separating males and females no division of the sample was undertaken. Instead, we used the entire sample as a test sample. Because we used this small participant pool to evaluate our model’s predictive accuracy and not to evaluate facial morphology or expressions, we retained those with beards.

Read also: PRP Facial Recovery Tips

Image preparation

To ensure that only the faces and not the use of a certain background or a certain set of clothes are biasing our results, we used the dlib library implemented in Python to locate and crop the face. In addition we used the Python OpenFace library to align the images such that faces are centered and have the same rotation.

Danish political alignment includes a color element (with left- and right-wing parties denoted by red and blue colors, respectively). To prevent color choices from influencing model results we used OpenCV to turn all photos into black and white images. An example image that has been cropped and manipulated as described is provided in Fig. 1. This cropping is all that was performed prior to our first analyses, though as described in the results section, we then found an additional cropping step (described further there) was required.

Measures

Facial expressions

We used the Face API from Microsoft Azure's Cognitive Services to identify facial expressions. Facial photographs are assigned a score for how much the face in question resembles each of a list of emotional states-e.g., happiness, sadness, surprise. Descriptive statistics for our full primary sample (Table 1) indicated that faces were overwhelmingly scored as indicating happiness (80%), with neutrality indicated as the second most frequent (19%). The remaining expressions were indicated to be infrequent (ranging of 0.00 to 0.01%). While this balance of facial expressions is not necessarily implausible for photos provided in an election context, the results could potentially reflect imperfect scoring procedures of the Face API.

Faces high in fWHR are typically perceived as aggressive and dominant and as having higher rank and status. High fWHR have been found to predict achievements in both the business and athletic domains.

fWHR scores are obtained by dividing the bizygomatic breadth (the length of the white box) by the distance between the top eyelid and the upper lip (the height of the white box). This dataset consists of 5500 faces rated by 60 raters using a beauty score ranging from 0-4. (Scoring attractiveness as the average value from multiple raters is commonly employed in political science studies on facial attractiveness.

To predict attractiveness scores for our study participants we employ the same CNN model that we used for ideology. The only difference between the network used for predicting ideology and this model is that we are faced with a regression problem instead of a classification problem. Consequently the loss-function is therefore different. We create one model for each gender, using only Caucasians. The model for females had a mean squared error (MSE) of 0.41 the model for males had an MSE of 0.35. Attractiveness scores in our sample averaged 3.35 [SD = 0.26] for males and 3.24 [SD = 0.28] for females on a scale from 0-4.

Analyses

#

tags: #amateur #public #facial #definition