ABSTRACT:
OBJECTIVE: The Patient Health Questionnaire-9 (PHQ-9) is a widely used self-report measure of
depression whose psychometric properties were found to be merely acceptable. Insufficient
attention has been devoted to its clinimetric validity, while its clinical utility
is still debated, particularly for assessing depression severity. This is the first
study to test the PHQ-9 construct validity and clinical utility based on clinimetric
principles.
METHODS: An online survey of 3,398 participants was conducted. Item response theory models
(Rasch and Mokken analyses) were used to assess the PHQ-9 validity and determine its
clinical utility.
RESULTS: Fit to the Rasch model was achieved after adjusting the sample size. Items 2, 4, 6,
and 9 over-discriminated, while items 1, 5, and 7 under-discriminated. Local dependency
between items 2 and 6 was indicated. The PHQ-9 was not unidimensional. A Loevinger's
coefficient of 0.49 was found, indicating an acceptable level of scalability.
CONCLUSION: The PHQ-9 is an instrument with potential clinical utility as an overall index of
depression, mainly for screening purposes. Substantial revision, particularly in the
wording of over- and under-discriminating items, is needed.
Keywords: PHQ-9; depression; patient-reported outcome measure; validity; Rasch analysis.
| Received: October 25 2023; Accepted: December 09 2023 |
INTRODUCTION
The scientific and clinical interest in patient-reported outcome measures (PROMs) has grown in recent decades.1 PROMs are self-report tools assessing data directly from patients about how they function or feel in relation to a health condition or treatment.2-4 Ideally, they were developed to assess symptom burden and disease severity, biopsychosocial functioning, quality of life, and well-being.4 A number of studies have examined the measurement properties of these assessment instruments, although most have applied a classical psychometric approach rather than clinimetric principles.5,6
Clinimetrics was originally introduced by Alvan R. Feinstein in 1980s7,8 as an innovative clinically-based evaluation method for a wide range of clinical issues not included in customary taxonomy.9-14 Over the decades, this scientific discipline has provided instruments for the identification, classification, and temporal distinction of clinical phenomena. The CLIPROM criteria, which consider clinimetric criteria relevant for PROMs, are a recent contribution.15 Unlike classical psychometrics, clinimetrics applies unidimensionality to the assessment of construct validity and uses it to: 1) evaluate whether each item of a rating scale covers unique clinical information, 2) test whether symptoms belong to an underlying syndrome, and 3) determine the extent to which the total score of a tool is a statistically sufficient measure of the severity of the investigated clinical condition.10,15,16
Several PROMs have been developed for depression, and their dimensionality has been largely documented from a psychometric point of view. Among them, the Patient Health Questionnaire-9 (PHQ-9)17,18 is a widely used self-report measure of depression in general19-21 and clinical populations (e.g., cardiology,22 dermatology,23 diabetology,24 gastroenterology,25 neurology,26,27 oncology,28 primary care,29-31 rheumatology,32 and psychiatry33-35). The psychometric properties of the PHQ-9 were considered good, particularly its internal consistency, test-retest reliability, and factorial validity.17,36-39 However, evidence is lacking about its clinimetric validity. In recent years, there has also been a debate about the clinical utility of the PHQ-9. Some authors recommended it as a dimensional assessment of depression,40,41 while others have reported measurement problems with the current version.42,43 In particular, Christensen & Sparle-Christensen42 found that the PHQ-9 was an overall misfit to the Rasch model, mainly because of its disordered item threshold (items 1, 3, 4, 5, 6, and 9) and local dependency between items 2 and 6. Thus, such controversial issues should be clarified.
The present study proposes the first clinimetric analysis of the PHQ-9. Based on CLIPROM criteria,15 its main aims were to examine its construct validity using both Rasch and Mokken analyses and to determine the PHQ-9 clinical utility.10,12,14,15
METHODS
Participants
Students enrolled at the University of Florence during the 2021-2022 academic year with an active institutional e-mail address were eligible for participation. No exclusion criteria were applied. Participation was voluntary and uncompensated. The age distribution of the participants was as follows: 18-20 years, 344 (9.9%); 21-25 years, 1,060 (30.6%); 26-30 years, 310 (8.9%); < 18 years, four (0.1%), and > 30 years, 183 (5.3%). Age was not reported by 1,497 students, since this information was not made compulsory.
Procedure
An invitation with a link to the online survey was sent on May 13, 2022 to the institutional email addresses of University of Florence (Italy) students. The online survey was kept active until May 30, 2022. A total of 3,464 students participated, who were instructed about the research protocol (i.e., a brief description of the study, the involved investigators, the duration and content of the survey, the assessment method and time required as well as a guarantee of anonymity). The students provided informed consent to participate via an online form. Due to the need to collect data anonymously, no strategies were applied to limit duplicate responses, although the length of the survey and its limited period of availability likely discouraged this practice. PHQ-9 data were collected from 3,398 participants and analyzed in the present paper. This study, called Mental Health Literacy among students (MATTERS), was supported by the European University for Well-being consortium via the 2021 second call for Seed Funding.
Measures
The PHQ-9 is a self-report measure for diagnostic, monitoring, and screening purposes, as well as for assessing depression severity.17,18 The instrument consists of two parts. The first section investigates nine depression symptoms listed in the DSM-IV: 1) lack of interest, 2) depressed mood, 3) sleeping difficulties, 4) tiredness, 5) appetite problems, 6) negative feelings about self, 7) concentration problems, 8) psychomotor agitation/retardation, and 9) suicidal ideation.17,18 Respondents are asked to indicate how often they have been bothered by any of these symptoms in the last 2 weeks.17,18 Each symptom-item is rated on a 4-point Likert scale ranging from 0 (Not at all) to 3 (Nearly every day). Because the second part of the scale assesses functional impairment caused by depression, it was not used in the present study and, thus, the total PHQ-9 score was based on the first section alone.17,18 Total scores of 5, 10, 15, and 20 represent cutoff points for mild, moderate, moderately severe, and severe depression, respectively.44 A cut-off of = 10 points showed a sensitivity of 88% and a specificity of 88% for major depression.17
Statistical analyses
The Rasch analysis was conducted using Rasch Unidimensional Measurement Models (RUMM2030) software45 to test the following clinimetric properties:- Overall fit to the model, which was evaluated using the chi-square item-trait interaction statistics.46,47 Such statistics provided a summary measure of how the PHQ-9 conforms to Rasch model expectations.48 A non-significant chi-square probability value indicated a good level of overall fit.46,47
- Individual item and person fit: standardized fit residual values for items and participants were examined for any indication of misfit.
- Dimensionality testing: principal component analysis of residuals was conducted to identify the two most different subsets of items (i.e., the most positively and negatively factor-loading items in the first component). Paired t-tests were then performed to compare scores on the item subsets. If more than 5% of t-tests were significant, the instrument would not be considered a unidimensional measure of depression.48,49
- The Person separation reliability index was assessed to estimate the clinimetric sensitivity of the PHQ-9, i.e., its ability to discriminate among respondents with different levels of depression.47,50
Mokken analysis, a non-parametric version of item response theory models,10,51 was performed to further assess the PHQ-9 dimensionality or scalability. Mokken analysis was conducted using Stata version 7. The Stata LoevH command was used to compute Loevinger's coefficients of homogeneity. According to Mokken,51 Loevinger's coefficients of homogeneity52 between 0.30 and 0.39 are considered acceptable, while a value = 0.40 is a clear demonstration of a rating scale scalability.10
Ethics statement
This study was approved by the University of Florence ethics commission (no. 184, November 23, 2021).
RESULTS
Overall and individual item fit to the Rasch model
Rasch model fit statistics are shown in Table 1. A significant item-trait interaction statistic (?2 = 356.00, degrees of freedom = 81, p < 0.001) was found, thus revealing an initial misfit to the Rasch model expectations (Table 1, Analysis 1). Misfit to the Rasch model remained after rescoring the disordered response categories (Table 1, Analysis 2). However, fit to the Rasch model was achieved after adjusting the sample size to 500 respondents (?2 = 42.22, degrees of freedom = 81, p = 0.999). The summary fit residuals for items and respondents were found to be within the acceptable limits of ± 2.5 (Table 1, Analyses 1-3). Table 2 shows the Rasch model fit statistics for individual PHQ-9 items. Items 2, 4, 6, and 9 over-discriminated, while items 1, 5, and 7 under-discriminated (Table 2).
| Sample | Analysis | Model fit (overall) | Item fit residual, (mean [SD]) | Person fit residual (mean [SD]) | PSI | Dimensionality,significant t tests (95%CI) | Local dependency(residual correlation> 0.20 above average) | Differential item functioning:age group, degree |
|---|---|---|---|---|---|---|---|---|
| All items | 1 | ?2(81) = 356.00, p < 0.001 | -0.41 (4.91) | -0.27 (0.95) | 0.85 | 7.09 (6.36-7.83) | Items 2 and 6 | Items 8 (age group) and 6 (degree) |
| Rescoring all items (0112) | 2 | ?2(81) = 286.94, p < 0.001 | -1.56 (3.69) | -0.39 (1.04) | 0.82 | 6.24 (5.51-6.97) | Items 2 and 6 | None |
| Adjusted sample (n=500) | 3 | ?2(81) = 42.22, p = 0.999 | -1.56 (3.69) | -0.39 (1.04) | 0.82 | 6.24 (5.51-6.97) | Items 2 and 6 | None |
PSI = person separation index (with extremes).
| Item | Location | Fit residual | ?2 | Probability┼ |
|---|---|---|---|---|
| 1. Little interest or pleasure in doing things | -0.687 | 4.859 | 11.929 | 0.217 |
| 2. Feeling down, depressed, or hopeless | -0.360 | -7.723 | 135.448 | 0.000 |
| 3. Trouble falling or staying asleep, or sleeping too much | -0.481 | 2.458 | 7.984 | 0.536 |
| 4. Feeling tired or having little energy | -1.312 | -4.287 | 51.839 | 0.000 |
| 5. Poor appetite or overeating | 0.061 | 5.171 | 22.584 | 0.007 |
| 6. Feeling bad about yourself - or that you are a failure or have let yourself or your family down | -0.051 | -5.697 | 54.322 | 0.000 |
| 7. Trouble concentrating on things, such as reading the newspaper or watching television | 0.065 | 3.226 | 14.399 | 0.109 |
| 8. Moving or speaking so slowly that other people could have noticed? Or the opposite - being so fidgety or restless that you have been moving around a lot more than usual | 1.240 | 1.922 | 20.443 | 0.015 |
| 9. Thoughts that you would be better off dead or of hurting yourself in some way | 1.526 | -3.635 | 37.048 | 0.000 |
┼ Bonferroni-adjusted at 1%.
Dimensionality and scalability
Significant t-tests outside the critical value of 5% were found for opposing residuals, indicating that the PHQ-9 was multidimensional (Table 1, Analyses 1-3). Mokken analysis showed that the total score had acceptable scalability, with Loevinger's coefficient of homogeneity being 0.49. As shown in Table 3, individual PHQ-9 items also showed acceptable scalability, with Loevinger's coefficients of homogeneity ranging from 0.44 to 0.56.
| Item | Mean score | Loevinger's coefficient of scalability |
|---|---|---|
| 1 | 1.45 | 0.44 |
| 2 | 1.36 | 0.56 |
| 3 | 1.47 | 0.48 |
| 4 | 1.76 | 0.54 |
| 5 | 1.17 | 0.45 |
| 6 | 1.22 | 0.53 |
| 7 | 1.16 | 0.46 |
| 8 | 0.55 | 0.44 |
| 9 | 0.39 | 0.52 |
Local dependency
Local dependency was detected between items 2 (Feeling down, depressed or hopeless) and 6 (Feeling bad about yourself - or that you are a failure or have let yourself or your family down).
Differential item functioning
Item 8 (Moving or speaking so slowly that other people have noticed - or the opposite - being so fidgety or restless that you have been moving around a lot more than usual) showed significant differential item functioning for age. Significant differential item functioning for degree was observed in item 6.
Person separation reliability index
Person separation reliability indices ranged from 0.82 to 0.85 (Table 1, Analyses 1-3), indicating that the PHQ-9 could be reliably used to distinguish between different groups of individuals.
DISCUSSION
The PHQ-9 was found to have potential clinical utility, despite requiring substantial improvement. Local dependency was observed, and several over- and under-discriminating items were found. Item 2 (Feeling down, depressed or hopeless) had the largest misfit to the Rasch model, probably because it is a compound question, which makes it difficult for respondents to provide precise answers. Because feeling depressed and feeling hopeless might be core symptoms of different clinical pictures,53 they should be investigated in separate items.
Item 6 (Feeling bad about yourself - or that you are a failure or have let yourself or your family down) had the same over-discrimination problem. Once again, this is a compound question referring to two different clinical dimensions.53-55 Such problematic over-discrimination might be the reason that the PHQ-9 overestimates depression severity, particularly when used as a dimensional measure.56-58 Indeed, it has been observed that, compared to other scales of depression severity assessment, the PHQ-9 is overinclusive in identifying patients with severe depression and underinclusive in identifying those with mild depression.58 The present findings consistently show that the PHQ-9 includes a mixture of over- and under-discriminating items. The challenge lies in establishing appropriate cut-off values, since setting them too low can result in numerous false positive diagnoses of depression, which might result in antidepressant overprescription.59,60
The present study also identified double- and triple-barreled questions,61-63 i.e., questions incorporating different clinical variables in a single item. For instance, item 9 (Thoughts that you would be better off dead or of hurting yourself in some way) combines two separate clinical entities (i.e., suicidal ideation and self-harm thinking) into a single question. This could bias responses and become a source of misfit. A rewording of double- and triple-barreled items is needed42,61,63,64 to improve the PHQ-9 clinical validity and clinimetric sensitivity in discriminating symptoms that might belong to different conditions (e.g., demoralization and depression) and distinguish between different levels of depression severity.12,15,65 Thus, one solution for item 9, for example, would be to split it into two subitems (Thoughts that you would be better off dead and Thoughts of hurting yourself in some way). The local dependency between items 2 and 6 (in line with Christensen & Sparle-Christensen42) can be solved by reformulating them. Item 2 might be split into two subitems (Feeling down and depressed and Feeling hopeless), while item 6 needs simplification (e.g., Feeling guilty or blaming yourself).
Paired t-tests of opposing residuals indicated that the PHQ-9 is multidimensional, while Mokken analysis showed that the items and the total score have acceptable scalability, which is an acceptable level of unidimensionality. This confirms the instrument's conflicting construct validity results, which have been reported in the literature.36,40-42,64,66-71 The PHQ-9 multidimensionality might be related to the fact that it was found to cover more than one dimension of depression severity. Each item measures a different depressive symptom (e.g., item 1 measures anhedonia, item 2 measures depressed mood, item 3 measures sleep problems, etc.), but all nine items measure the same underlying construct, depression, which explains the acceptable level of unidimensionality in the total score.
Therefore, future studies are needed, although it should be noted that the Person separation reliability indices were acceptable and that the PHQ-9 fit Rasch model expectations after adjustment for sample size. There is, thus, empirical justification for using the PHQ-9 as an overall index of depression, particularly when supplemented by other clinimetric indices, such as the Major Depression Inventory50,72,73 and the 6-item version of the Hamilton Rating Scale for Depression,74,75 which were found to be unidimensional. This is in line with the proposed use of the PHQ-9 as a screening, rather than a severity, measure.66,68
The present findings should be interpreted in light of some limitations. First, participants were recruited by convenience sampling, thus limiting generalizability of results. Future research using a sample of patients with depression is needed. Second, due to the cross-sectional design, the PHQ-9 incremental and predictive validity could not be evaluated. Third, no other measures of depression were used. In future investigations, depression should be assessed with other PROMs to test the PHQ-9 clinical and concurrent validity.
The present clinimetric analysis suggests that the PHQ-9 can be used as an overall index of depression, mainly for screening purposes. Substantial revision, particularly in item wording, is needed to improve its construct validity and clinical utility.
ACKNOWLEDGEMENTS
This study was supported by the EUniWell (European University for Well-being) consortium via the 2021 second call of Seed Funding.
DISCLOSURE
The authors report no conflicts of interest.




PDF
Print
Send this article by email
How to cite this article
Submit a comment
Mendeley
Pocket




