Likert vs. Q Sort Scales

When psychologists want to gather information from a patient or an individual, they will most likely resort to questionnaires, particularly self-report tests. The most common question format of those questionnaires is the five options answers, ranging from “extremely disagree” to “extremely agree”. This is called the Likert format and has been used so far because it is effective and we have the statistical tools to understand it (Joshi et al., 2015). Despite its extended use, it would be a mistake to think that the Likert scale is the only way of access into human psychology. Given its limitations, the fact that there are alternatives is a good thing (Bishop & Herron, 2015). This article sets to explain the Likert scale along with its limitations. An alternative to the Likert scale is also going to be introduced. This alternative approach complements the Likert approach and, if properly used, would bring many more insights to the psychological science.

Figure 1: Rensis Likert

The Likert Scale is a psychometric tool for measuring many types of psychological information. A Likert question is used when a statement is presented to a respondent, and the respondent has to indicate the degree of agreement or disagreement they have with the statement in a multiple-choice format (Tullis & Albert, 2013). For example, in the HEXACO personality test one statement is: “On most days, I feel cheerful and optimistic”. The respondent has to —in respect of their own experience— answer this statement by choosing one of five options: strongly disagree, disagree, neutral, agree, and strongly agree (Ashton & Lee, 2007).

Named after Rensis Likert, the Likert scale is organized on a continuum and it best serves its purpose with ordinal data. Generally speaking, there are three types of data in books of statistical analyses: categorical, ordinal, and continuous. An example of categorical data would be sex or means for public transportation, and a Likert scale is of no use for this type of data. The ordinal data can be seen in socioeconomic status, education level, or satisfaction rating. Using Likert scales is the best option for this data because no other type of data can be organized in a continuum. A type of ordinal data, known as rank-ordered data, is seen in sports when teams or players are ranked, or in politics when candidates' popularity is ranked. For rank-ordered data, a different scale is used and it is explained further down in the article. Continuous data are sometimes organized by a Likert scale but some voices have raised against this practice. Examples of continuous data include height, weight, and times in a race (Dettori & Norvell, 2018).

Figure 2: 1 to 10 by xkcd (xkcd, 2011)

Some advantages of a Likert scale are that it is generalized precisely because it is easily understood, the responses are easily quantifiable and many statistical analyses can be applied to this kind of data. It allows for more nuances to the questions than yes or no answers, and it also allows the expression of neutral responses. On more hands-on issues, the Likert scale is quick, efficient, inexpensive, easily applicable both offline and online, and versatile (Jamieson, 2004).

However, the Likert scale has its problems. It is clear that, for example, “extremely agree” scores higher than a plain “agree” or “somewhat agree”, but we cannot be sure about the meaning in the spaces between the answer options. The distance between “agree” and “somewhat agree”, and “somewhat agree” and “extremely agree” may not be the same (Jamieson, 2004). To have a clearer idea of this problem, one can think about the places of runners in a competition. The difference between the first and the second runner could be counted in seconds whereas the difference between the second and the third could be five times that of the previous pair. It is clear who is the winner and who arrived first, but this information is just half of the picture.

Two additional problems with the Likert scale are the influence the questions have over each other and the avoidance of extreme answers. Concerning the former, one of the assumptions of psychological questionnaires is that the answer given to one question is independent of the answers given to any other question in the test (Houts & Edwards, 2015). Sometimes, however, Likert scales have questions which are very similar. If the HEXACO personality test, for example, asks first “Do you feel optimistic and cheerful on most days?”, and then asks “Are you optimistic about life?”, chances are any respondent would answer the second question similar to the first, though their meaning is dissimilar.

Figure 3: Data analysis by Brad Yeo

Regarding the avoidance of extreme answers, respondents tend to avoid choosing one extreme or the other. For example, in the Left Wing Authoritarianism scale (Costello et al., 2022), two statements are: “The rich should be stripped of their belongings and status” and “Rich people should be forced to give up virtually all of their wealth”. Now, most people —due perhaps to social desirability— would probably choose intermediate responses like “somewhat agree” or “neither agree nor disagree” which, in this case, do not reflect the true motivation of the respondents and provide a false representation of what is truly happening.

There is a different way to obtain a more accurate response to a person's motivations and thinking patterns. It consists of ranking a population of heterogeneous items along a simple dimension from “most agree” to “most disagree”, or “most characteristic” to “most uncharacteristic” (Watts & Stenner, 2012). This is called the Q sort distribution and consists of a distribution of, commonly, 9 to 13 ranking values. The items placed in the middle of the distribution score 0, whereas the items in the extreme of, for instance, the 9 ranking values Q sort, score +4 and -4. In one example of a Q sort distribution —the Riverside Situation Q sort (Funder et al., 2000)—, the respondents had to rank 89 items in 9 bins. The ranking of the items, however, is not completely free. Each bin has constraints on the total number assigned to them. This constraint is arranged in such a way that the final shape of the distribution of items would resemble that of a quasi-normal distribution —in other words, a bell-shaped distribution. In the case of 89 items, the distribution would go as follows: 3 on each extreme, 6 in the following bins, 11 in the next ones, 15 next, and in the middle 19 items. Due to this constraint imposed on the distribution of items, these kinds of measurements are also called forced-choice measurements.

Figure 4: Shopping for Society? Consumers’ Value Conflicts in Socially Responsible Consumption Affected by Retail Regulation, by (Lee et al., 2017)

Besides the difference in form and structure, the Q sort has particular characteristics that a Likert scale lacks. In a Q sort, the primary purpose is to discern people’s perceptions of their world from their point of self-reference, what is called subjectivity in psychology. This is attained by asking the respondent to rank the items having in mind the meaning of each of them in relation to the rest of them. Whereas in a Likert scale scenario, the respondent could assign the same value to all the items if it comes to that, in a Q sort, the respondent has to perform a task of classification and organization (Watts & Stenner, 2012)s.

A job recruiter would have a better representation of a candidate’s personality if between two questions that convey positive meaning like: “Are you responsible?” and “Are you creative?”, instead of getting two 5 scores from a Likert scale, the respondent provides a rank in which creativity is more important than responsibility.

Once the responses are collected, the procedure is similar to a Likert scale but the emphasis is different. While the two of them require correlations, the correlations of the Likert scale are performed among the items whereas the correlations in a Q sort are performed among people. In other words, the Likert scale compares items while the Q sort compares people. The final product allows identifying patterns of response across individuals and forming groups of similar response styles (Gao & Soranzo, 2020).

Figure 5: Grader Types by PhD Comics (n.d)

The Q sort is the way of knowing more about subjectivity objectively. It reveals the hidden meaning and understanding of the respondent's world. This approach concurs with the quantum conclusion:

We have to remember that what we observe is not nature in itself but nature exposed to our method of questioning (Heisenberg, 2007, p. 24 - 25).

This has even more relevance in the field of psychology where most of the things that affect an individual’s life, first go through the perception of them.

Figure 6: Being a Writer by Joey Guidone

There is much research done on both the Likert and Q sort answering methods that surpasses the scope of this article. However, we have presented the most important concepts of these methods. The Likert scale has served psychology —and many other sciences as well— to a high degree and most of the scientific discoveries done in psychology have been obtained by using it. It is time, though, to integrate new aspects of the human experience in the study of psychology, and the Q sort —along with others— can help with this aim. In the end, the conclusions that psychology would open itself to will probably revolutionize the field in ways that one can only imagine.

Bibliographical Resources

Ashton, M. C., & Lee, K. (2007). Empirical, Theoretical, and Practical Advantages of the HEXACO Model of Personality Structure. Personality and Social Psychology Review, 11(2), 150–166. https://doi.org/10.1177/1088868306294907

Bishop, P. A., & Herron, R. L. (2015). Use and Misuse of the Likert Item Responses and Other Ordinal Measures. International Journal of Exercise Science, 8(3), 297–302. Costello, T. H., Bowes, S. M., Stevens, S. T., Waldman, I. D., Tasimi, A., & Lilienfeld, S. O. (2022). Clarifying the structure and nature of left-wing authoritarianism. Journal of Personality and Social Psychology, 122(1), 135–170. https://doi.org/10.1037/pspp0000341 Dettori, J. R., & Norvell, D. C. (2018). The Anatomy of Data. Global Spine Journal, 8(3), 311–313. https://doi.org/10.1177/2192568217746998 Funder, D. C., Furr, R. M., & Colvin, C. R. (2000). The Riverside Behavioral Q-sort: A tool for the description of social behavior. Journal of Personality, 68(3), 451–489. https://doi.org/10.1111/1467-6494.00103 Gao, J., & Soranzo, A. (2020). Applying Q-Methodology to Investigate People’ Preferences for Multivariate Stimuli. Frontiers in Psychology, 11. https://www.frontiersin.org/article/10.3389/fpsyg.2020.556509 Heisenberg, W. (2007). Physics and Philosophy: The Revolution in Modern Science. HarperCollins. Houts, C. R., & Edwards, M. C. (2015). Comparing Surface and Underlying Local Dependence Levels via Polychoric Correlations. Applied Psychological Measurement, 39(4), 293–302. https://doi.org/10.1177/0146621614561492 Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38(12), 1217–1218. https://doi.org/10.1111/j.1365-2929.2004.02012.x Joshi, A., Kale, S., Chandel, S., & Pal, D. (2015). Likert Scale: Explored and Explained. British Journal of Applied Science & Technology, 7(4), 396–403. https://doi.org/10.9734/BJAST/2015/14975 Tullis, T., & Albert, B. (2013). Chapter 6—Self-Reported Metrics. In T. Tullis & B. Albert (Eds.), Measuring the User Experience (Second Edition) (pp. 121–161). Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-415781-1.00006-6 Watts, S., & Stenner, P. (2012). Doing Q Methodological Research: Theory, Method & Interpretation. SAGE.

Visual sources

Figure 1: “ISR, Rensis Likert, December 1961; BL005652.” University of Michigan Library Digital Collections. Accessed: July 09, 2022. Retrieved from:

https://quod.lib.umich.edu/b/bhl/x-bl005652/bl005652.

Figure 2: Xkcd. (2011). 1 to 10. [Cartoon]. Retrieved July 8, 2022, from: https://xkcd.com/953/

Figure 3: Yeo, B. (n.d). Data [Picture]. Retrieved from: https://www.theispot.com/stock/yeo

Figure 4: Lee, J.-M., Kim, H.-J., & Rha, J.-Y. (2017). Shopping for Society? Consumers’ Value Conflicts in Socially Responsible Consumption Affected by Retail Regulation. [Graphic]. MDPI. Retrieved from:

https://www.mdpi.com/sustainability/sustainability-09-01968/article_deploy/html/images/sustainability-09-01968-g002-550.jpg

Figure 5: PhD Comics. (n.d.). Grader Types. PhD Comics. [Cartoon]. Retrieved July 8, 2022, from https://phdcomics.com/comics.php?f=1404

Figure 6: Guidone, J. (2017) Being a Writer [Illustration]. Retrieved from:

https://www.behance.net/gallery/56026413/Being-A-Writer