Data

The dataset consists of manual transliterations of audiovisual recordings generated by 404 YouTube vloggers that explicitly show themselves in front of a webcam talking about a variety of topics including personal issues, politics, movies, and books. The corpus was originally presented in ^{Biel et al. (2013)}. There are no content-related restrictions in the videos, and the language is natural, diverse, and informal. The transliterations contain approximately 10 000 unique words and 240 000 word tokens. The data is gender-balanced (52% female). The transliterations were originally produced in the English language, and the videos in the database were automatically labeled according to the five traits of the OCEAN model. The labeling process was performed using the Amazon Mechanical Turk (^{Buhrmester etal., 2016}) and the Ten-Item Personality Inventory (^{Gosling et al., 2003}). Figure 2 shows histograms with the scores assigned to each trait. Some statistical information on the scores is also provided in Table 1.

Source: Authors

Figure 2 Histogram for the score in the five traits  

Table 1 Statistical information of the personality scores 

Med.: Median; Min.: Minimum; Max.: Maximum; T1: 1st tertitle; T2: 2nd tertile; Var.: Variance.

Source: Authors

Pre-processing

Before feature extraction, the data need to be cleaned and standardized in order to remove the 'noise' and prepare them for analysis. The steps followed during the text preprocessing include: i) removal of non-content words like 'xxxx', 'um', 'uh', and others; ii) conversion of all texts to lower case, removal of punctuation, numbers, and stop-words; and iii) lemmatization, which is applied to transform words into their root form. Figure 3 shows the number of words per text before and after pre-processing.

Source: Authors

Figure 3 Number of words per text before and after pre-processing  

Feature extraction

One of the goals of Natural Language Processing (NLP) is to mathematically represent words of a text in a vector space. This vector representation is such that similar words are represented by nearby points. In this work, we consider three different techniques to create said vectors: Word2Vec, Global Vectors (GloVe), and BERT. Details of each model are presented below.

Word2Vec: Word2Vec considers information from nearby words to represent target words with a shallow neural network whose hidden layer encodes the word's representation (^{Mikolov et al., 2013}). The network contains one hidden layer, whose dimension is equal to the embedding size, which is smaller than the one at the input/output vector. A softmax activation function is applied at the output layer, so that each element of the output vector represents the probability of a given word appearing in the context. Word embeddings are obtained from the hidden layerfollowingthe Continuous Bag of Words (CBoW) method, which considers information from the neighboring words to model the probability of the target word appearing in a given context. In this case, the input is the words around in the given context and the output is the target word. For example, assuming a context of 2 and 'drink' as the target word, in the sentence 'I will drink orange juice', the input would be 'I', 'will', 'orange', and 'juice', while the output would be 'drink'.

All inputs and output data have the same dimension and are coded as one-hot vectors. The length of the vector is equal to the size of the vocabulary in the corpus, which considers unique words. Typically, these unique words are coded in alphabetical order, that is, one-hot vectors for words beginning with 'a' are expected to have the target '1' in a lower index, while words beginning with 'z' are expected to have the target '1' in a higher index. Figure 4 shows the structure of the network for the CBoW method, where W Є ℝ ^VxN refers to the weight matrix that maps inputs Xi to the hidden layer, and W Є ℝ ^NxV is the weight matrix that maps outputs of the hidden layer to the final output layer. The neurons in the hidden layer copy the weighted sum of inputs to the next layer (i.e., linear activation function).

Source: Modified from ^{Bellei (2018)}

Figure 4 CBoW topology used in Word2Vec. x: one-hot vectors; V: size of the vocabulary; C: number of context words; h: hidden layer with N neurons, where N is also the number of dimensions to represent the word; y: one-hot vector for target word.  

GloVe: The Global Vectors for Word Representation (GloVe) model creates word vectors by examining the cooccurrences of words within a corpus (^{Pennington et al., 2014}). Before the model is trained, an X co-occurrence matrix is created, where each element X_ij represents the frequency for the i-th word to appear in the context of the j-th word. The corpus is traversed only once to create the X matrix, and these co-occurrence data are then used instead of the corpus. Once X is created, the task is to generate the vectors in a continuous space for each word of the corpus. Vectors with a smooth constraint will be produced for each pair of words (w _i , W _j ):

where w _i and W_j are word vectors and b_i and b_j are scalar biases associated with the i-th and j-th word, respectively.

This model creates word vectors with relevant information about how each pairof words coexist. The objective function of the optimization problem is called J, and it evaluates the mean square error of Equation 1, weighted by the function f:

From Equation 2, f (X_ij) is chosen in such a way that common word pairs are not considered (those with large X_ij values) because they would deviate too much from regular words:

In Equation 3, x_max refers to the maximum co-occurrence value between the i-th and j-th word. When common word pairs are found (such that X_ij > x_max), the function limits its output to 1. For all other word pairs, a weight in the range 0 - 1 is computed. The distribution of weights depends on a, which is a hyper-parameter that controls the sensitivity of the weights to increased co-occurrence counts.

BERT makes use of transformers, which are attention mechanisms that learn contextual relations between words (or sub-words) in a text (^{Devlin et al., 2018}). In its general form, a transformer includes two separate mechanisms: an encoder that reads the text input and a decoder that produces a prediction for the task. BERT allows both left and right contexts to influence many language representations that include word predictions (^{Devlin et al., 2018}). To effectively train a bidirectional transformer, BERT uses two techniques called Masked Language Model (MLM) and Next Sentence Prediction (NSP). The transformer architecture comprises a stack of encoders and a stack of decoders, where the encoders are composed of a self-attention layer and a Feed-Forward Neural Network (FFNN). Encoders are identical in structure and are connected to decoders, which include all the elements present in an encoder, in addition to an encoder-decoder attention layer between the self-attention layer and the feed-forward layer. Figure 5 shows the architecture of the transformer in BERT.

Source: Modified from ^{Alammar (2018)}

Figure 5 Architecture of the transformer used in BERT.T.E: text embedding; S.E: segment embedding; P.E: positional embedding.  

Pre-trained models were used for Word2Vec, GloVe, and BERT. For Word2Vec and GloVe, we considered the Python gensim module (^{Rehurek and Sojka, 2010}). In the case of Word2Vec, the word2vec-google-news-300 model was employed, which was trained with Google News, with a corpus of around 100 billion words and a vector dimension of 300 (^{Mikolov, 2015}). For GloVe, the model glove-wiki-gigaword-300 was employed, which was trained with the Wikipedia 2014 + Gigaword 5 corpus. In this case, there are 5,6 Billion tokens, a vocabulary of 400 000 words, and a vector dimension of 300 as well (^{Pennington, 2014}). To obtain word embeddings based on BERT, we used the WEBERT Toolkit (^{Perez, 2020}), which is a Python tool typically used to obtain pre-trained BERT embeddings in English and Spanish. To train the BERT word embeddings in the English language, the Multi-Genre Natural Language Inference corpus was used. We considered two different pre-trained BERT models: (i) BERT-base and ii) BERT-large, where the last layer (768 units for BERT-base and 1 024 units for BERT-large) was taken as the word-embedding representation.

Vector representations for each transcript are created per word, so that x Є ℝ ³⁰⁰ for Word2Vec and GloVe, x Є ℝ ⁷⁶⁸ for Bert-base, and x Є ℝ ¹⁰²⁴ for Bert-large. Since texts correspond to spontaneous speech, they have a different number of words per individual. To obtain vectors of a fixed dimension per text, six statistics are computed along the vectors: mean, standard deviation, skewness, kurtosis, minimum, and maximum. The resulting feature matrix is given by X Є ℝ ^Nx1800 for Word2Vec and Glove, X Є ℝ ^Nx4608 for BERT-base, and X Є ℝ ^Nx6144 for BERT-large. Where N is the number of transliterations (N = 404).

Classification and regression models

Since Support Vector Machines (SVM) are one of the most used classification methods in the state of the art, and considering their robustness regarding high-dimensional representation spaces (^{Scholkopf et al., 2002}), we decided to adopt this as our main framework for the classification and regression experiments. The following subsections provide some details of the mathematical background of the methods. However, we recommend that the reader refer to ^{Scho lkopf et al. (2002)} for a more comprehensive description of the methods.

Bi-class classification: In this case, the goal is to discriminate data samples by finding a separating hyper-plane that maximizes the margin between classes. Soft-margin SVMs allow errors in the process of finding the optimal hyper-plane. These allowed errors are data samples located on the wrong side of the hyper-plane but within the optimal margin. An example of a soft-margin SVM is shown in Figure 6.

Source: Modified from ^{Dey (2018)}

Figure 6 Soft-margin SVM  

The decision function of a soft-margin SVM is expressed according to Equation 4, where ξ_n is a slack variable that penalizes the number of errors allowed in the optimization process; y_n Є {-1, +1} are class labels; ϕ (x_n) is a kernel function that transforms the feature space x into a higher dimensional space, where a linear solution to the problem can be found; and the weight vector w and the bias b define the separating hyperplane.

The optimization problem is defined in Equation 5, where the hyper-parameter C controls the offset between x_n and the margin width. Samples xn that satisfy the condition of equality in Equation 4 are called 'support vectors' (x_m).

Tri-class classification: We adopted the One vs. All approach (OvA, also called One vs. Rest, OvR). This method consists of building one SVM per class, which is trained to distinguish the samples of one class from the samples of all the other classes. The decision is made according to the maximum output among all SVMs (^{Milgram et al., 2006}). This approach requires that each model predict a probability score per class. The max argument of these scores (class index with the highest score) is then used to predict a class.

Regression: We implemented a Support Vector Regressor (SVR) to predict the value of each label assigned to the personality traits. An SVR is an extension of an SVM, where, instead of integer-valued labels, real-valued labels are predicted. Particularly, an ε-SVR aims to find a linear function f (x) where only samples outside the ε-radius 'tube' are penalized (^{Smola and Schölkopf, 2004}) (Figure 7). The linear regression function f(x) is represented in Equation 6 as:

Source: Modified from ^{Smola and Schölkopf (2004)}

Figure 7 Linear SVR with ε-insensitive loss function 

where (•, •) denotes the dot product and b Є ℝ. The resulting optimization problem is written as follows:

The assumption in Equation 7 is that the function f(x) exists and that the convex optimization problem is feasible. However, this is not always the case. Thus, similarly to the soft-margin SVM, one can introduce slack variables ξ_i and ξ_i* to cope with otherwise infeasible constraints of the optimization problem in Equation 7. The resulting optimization problem is as follows (^{Vapnik, 1995}):

In Equation 8, the constant C > 0 determines the tradeoff between the flatness of f and the maximum allowed deviation ε. This corresponds to the so called ε-insensitive loss function |ξ|_ε, which is described in Equation 9:

Figure 7 illustrates the concept. Note that only points outside the region between the dotted line (the 'tube') contribute to the cost. Deviations are linearly penalized, although it is possible to extend SVM to nonlinear functions (^{Smola and Schölkopf, 2004}; ^{Ranković et al., 2014}).

Parameter optimization: For both SVM and SVR, Gaussian kernel and linear kernel were considered in our preliminary experiments. However since the results with the first one were better in most of the cases, only results with the Gaussian kernel are reported (see Appendix with results for the linear kernel). The hyper-parameters C, y, and £ were optimized through a grid-search up to powers of ten between 1 X 10^-4 and 1 X 10⁴. A subject-independent 10-fold Cross-Validation (CV) strategy was followed in the training process, i.e., the data were divided into 10 groups (randomly chosen, but data from the same subject were never included in the train and test fold simultaneously). The CV strategy was repeated 10 times to evaluate the generalization capability of the model (^{Kohavi, 1995}). The reported metrics correspond to the average and standard deviation of the ten repetitions.

Performance evaluation

The classification systems used in this work were evaluated with typical performance metrics including the confusion matrix (^{Powers, 2020}). For the particular case of a two-class classification problem, the matrix is as shown in Table 2.

Table 2 Confusion matrix 

Source: Authors

According to this matrix and taking class 0 as the target one, the following terms are defined:

Derived from the aforementioned terms, different performance measures are defined and taken into account, including accuracy (ACC), sensitivity (SEN), specificity (SPE), and the Fl-score (F1). Apart from the aforementioned measures, the receiver operating characteristic curves (ROC) were used as a graphical representation that summarizes the performance of binary-classification systems. The performance of the multi-class classification systems was evaluated with the Unweighted Average Recall (UAR) and Cohen's kappa coefficient (k).

The regression systems of this study were evaluated according to Spearman's correlation coefficient (p).

Additionally, to allow comparisons with different works in the literature that report results with different measures, we decided to include other metrics such as the Mean Absolute Error (MAE), the Root Mean Square Error (RMSE), and the coefficient of determination, R².

Data distribution and statistical analyses

The personality trait estimation considers the values assigned to each sample for each trait. Their distribution per trait is presented in Figure 2, and the corresponding statistics are presented in Table 1. For the binary classification scenario, the scores of each trait are divided around their median, i.e., samples with values below the median are considered weak, while those above are labeled as strong. This distribution criterion allows for a balanced number of samples per trait. The median threshold is shown in Figure 8 as a red dotted line. The distribution of the data for the tri-class classification problem is made according to the tertiles of the distribution of the scores per trait. This strategy guarantees balance among the three resulting subgroups. The distribution of these three subgroups is shown in Figure 8 as the three shadowed regions. The number of samples per class and subgroup (two for the bi-class problem and three for the tri-class problem) is summarized in Table 3.

Source: Authors

Figure 8 Score thresholds for the bi-class and tri-class classification problems. LP: low presence; MP: medium presence; HP: high presence.  

Table 3 Number of subjects for the bi-class and tri-class classification problem 

Source: Authors

Two statistical tests were performed. The first one was the Kruskal-Wallis test regarding the feature matrices extracted per sample and trait. This test was performed for the two scenarios: weak vs. strong presence of each trait, and the three levels of manifestation of the traits. In all cases, the null hypothesis was rejected with p << 0,01. The second statistical test aimed to evaluate whether the gender of subjects biased the distribution of the extracted features. We performed x² tests for both the bi-class and the tri-class scenarios. A possible bias regarding gender was discarded for the 'extraversion', 'conscientiousness', 'emotional stability', and 'openness to experience' traits, whereas a possible bias was found for the 'agreeableness' trait.

Experiment 1: personality trait estimation

This experiment was mainly based on SVR systems with Gaussian kernel. Experiments with linear kernel were also performed, and the results can be found in the Appendix. As mentioned before, to allow comparisons with respect to other works in the literature that use the same corpus as this study, the results are reported in Table 4 regarding the four metrics mentioned in the performance evaluation subsection, where the results in bold letters correspond to the best for each trait. Note that, in three out of the five traits, the best result was obtained when merging the Word2Vec and GloVe embeddings, except for the 'extraversion' trait, whose best result was obtained with the BERT-base embeddings, and 'openness to experience' whose best result was obtained with GloVe embeddings. When observing R ² and p, the best result was obtained for the 'agreeableness' trait (R² = 0,24 and p = 0,43), followed by 'conscientiousness' (R² = 0,16 and p = 0,41). The worst result was for the 'openness to experience' trait, with R² = 0,05 and p = 0,21. The lowest MAE = 0,55 was obtained with 'conscientiousness', and the lowest RMSE value was obtained with 'openness to experience', with a value of RMSE = 0,70.

Table 4 Results for personality trait estimation 

Fusion: Word2Vec + GloVe; BERT-b: BERT base; BERT-l: BERT large.

Source: Authors

Experiment 2: weak vs. strong presence of each trait

In this case, SVM classifiers with Gaussian kernel were used, and the results are reported in Table 5, where the results in bold letters correspond to the best for each trait. Note that four out of the five traits (all except 'openness to experience') obtained the best result when considering word embeddings based on BERT, three of them with the base model ('extraversion', 'agreeableness', and 'emotional stability') and the remaining one ('conscientiousness') with the large model. Furthermore, note that three out of the five traits exhibit accuracies above 60%. The best result was obtained for 'extraversion', with an accuracy of 64,7%, followed by 'agreeableness', with an accuracy of 64,3%, and 'conscientiousness', with 63,9%. As seen a few lines below, these results are similar to most of the results reported in the literature. The results also show that there is no a clear model that leads to the best results. This means that there is still a lot of work to be done in this field, which, apart from the challenge of extracting information from text, imposes an additional constraint due to the consistency of the labels, i.e., the evaluation of personality is very hard task for both humans and machines.

Table 5 Results for bi-class system: weak presence vs. strong presence of the trait 

Fusion: Word2Vec + GloVe, BERT-b: BERT base; BERT-l: BERT large.

Source: Authors

Results are shown more compactly in Figure 9, where the ROC curves resulting from the bi-class experiments are included. Each panel in the figure includes the results obtained with the three feature extraction approaches. The AUC values show that, in the majority of the cases, better results are obtained for 'extraversion' and 'conscientiousness', and also that the best AUC values are obtained by the Fusion of Word2Vec and GloVe embeddings, as well as by embeddings based on BERT.

Source: Authors

Figure 9 ROC curves obtained with Word2Vec, GloVe, Fusion (Word2Vec + GloVe), BERT-base and BERT-large embeddings  

Experiment 3: classification of personality traits into three levels

Three groups were created according to the scores in the personality traits, as it was explained in the data distribution and statistical analyses subsection: LP (low presence), MP (medium presence), and HP (high presence). The results of the tri-class classification are presented in Table 6 in terms of accuracy, F1-score, UAR, and k, where the results in bold letters correspond to the best for each trait. Note that, in four out of the five traits, the best results were obtained with the Fusion of Word2Vec and GloVe embeddings. Only in the case of 'conscientiousness' was the best result obtained with Word2Vec. It can also be observed that the models trained with BERT embeddings did not improve the performance of the models in comparison with the classical word embeddings (Word2Vec and GloVe) in any of the OCEAN traits. However, with a close look at the numbers, one can notice that the difference among different approaches is not that high, and the results are similar across different traits, i.e., between 40 and 46%.

Table 6 Tri-class classification results 

Fusion: Word2Vec + GloVe; BERT-b: BERT base; BERT-l: BERT large.

Source: Authors

To facilitate a detailed look at the results, confusion matrices are presented in Table 7, where, again, the results in bold letters correspond to the best for each trait. The results do not show a clear pattern when comparing the three trait levels. One would expect to see a relatively clear separation between LP and HP samples. However, in all cases, the target class shows the largest percentage, but the remaining portion is almost always equivalently distributed between the other two classes. Even though the models presented in this study could be improved, while also acknowledging that the addressed problem is very challenging, the behavior of the observed results may reflect possible labeling problems. We believe that there is a big gap in the study of personality traits based on linguistic patterns, which makes it necessary to work on collecting and labeling data considering the knowledge of expert psychologists and psycholinguists (^{Sun et al., 2019}; ^{Guan et al.,2020}).

Table 7 Confusion matrix for the classification of personality traits into three levels (results in %) 

Fusion: Word2Vec + GloVe; BERT-b: BERT base; BERT-l: BERT large.

LP: Low presence; MP: Medium presence; HP: High presence.

Source: Authors

The best results obtained throughout the regression and classification experiments are summarized in Figure 10. The first column of sub-figures shows the regression results. It can be observed that the regressor is doing a good job, especially in the first three traits, where Spearman's correlation is above 0,35. In the second and third columns, the resulting representation spaces from the bi-class and tri-class scenarios are shown, respectively. Note that, in these two cases, the Figures illustrate the result of applying a dimensionality reduction based on Principal Component Analysis (PCA). In the bi-class scenario, the hyper-planes shown in the Figure correspond to those found with the optimal parameters of the SVM, i.e., using the parameters reported in Table 5. Notice the high dispersion of the samples along the representation space. This is one of the reasons for the low accuracies found in the classification experiments. Finally, the tri-class scenario is shown in the third column of sub-figures, where three different colors are used to represent the three classes (LP, MP, and HP). These are also representations resulting from the PCA projection of the feature space. Even though the results appear to be low, the representation spaces show that the three sub-groups are found. These results, as well as the summarizing Figures, motivate us to continue working on this topic considering other approaches from fields such as language features and deep learning.

Source: Authors

Figure 10 Graphical summary of the best results  

Comparison with recent works

The results reported in this study were compared with respect to different works in the state of the art. We did not find any study working on the tri-class classification problem, so comparisons are only reported for regression and the the bi-class results.

The summary of our regression results and those reported by others in the literature are included in Table 8. According to the average results reported in the last row of the table, our approach shows similar performance to others reported in the literature. Our results are in fact better with regard to the MAE value in about 0,02. Now, in the case of RMSE, our results are 0,03 below (worst) and 0,04 below (worst) with respect to the previous result reported for R². Although other works in the literature did not report results for Spearman's correlation, we decided to examine it because this measure is more intuitive, especially when the 'actual vs. predicted' plot is shown (Figure 10).

Table 8 Comparison of our regression model with recent works 

Source: Authors

The comparison regarding the bi-class classification scenario is reported in Table 9. Note that, on average, our results are slightly better than the other works, except for those obtained by ^{Das-Kumar and Das-Dipankar (2017)}. Unfortunately, the authors of that work only report the average accuracy along with the five traits, which does not allow for direct comparisons in specific traits. If we consider the average performance in terms of the F1-score compared to the performance reported by ^{Salminen et al. (2020)}, we were able to improve the performance by 6,5%, which is relevant considering that we did not use neural networks. This gives us an idea that, for certain traits ('agreeableness', 'conscientiousness', and 'emotional stability'), classical methods such as those used in this paper (SVM with Gaussian kernel) yield better results than those found with other methods like the one reported by ^{Salminen et al. (2020)}.

Table 9 Comparison of our bi-class classification model with recent works 

Source: Authors