Measuring Impairment: Consistency of dichotomous and scaled responses to the six-question disability set

Background

Measuring impairment accurately is important for research and practice. Currently many researchers and practitioners depend on the six-item measure of impairment featured in the American Community Survey (ACS), which was defined and implemented in 2008 (see Brault, 2009 for a review of the development of this measure). Although this measure is widely used, responses to the items are not stable over time (Ward, Myers, Wong & Ravesloot, 2017).  This study examined the reliability of these items when measured dichotomously or with a scaled response option.

Methods

In 2016, respondents of the 3rd wave of the Ecology of Rural Participation Survey were asked early in the survey to respond “yes” or “no” to the standard six questions measuring impairment regarding hearing, sight, cognitive abilities, physical abilities, performing daily tasks like bathing and dressing, and running errands. In an effort to better understand the range of difficulty experienced by people regarding impairment, later in the survey respondents were invited to rate how difficult these same conditions have been for them in the past 30 days using a 0-10 scale (i.e., 0 = no difficulty and 10 = extreme difficulty). We expected those who reported impairment on the forced-choice question to report some difficulty on the scaled items.

Results

Respondents were very reliable in rating at least some difficulty with an impairment when they had answered “yes” to the forced-answer question (see Table 1). Generally, there is more variation in their reported levels of impairment when respondents endorsed an item compared to when they did not (see Figures 1 and 2).

Table 1. Percentage of respondents reporting “no” and “yes” to forced-choice question and more than zero on impairment scale

This table shows the percent of respondents who reported a 1 or higher on the difficulty scale for each impairment type. Each impairment type is subdivided into those who reported “no” or yes” for that impairment. Total n varies due to variability in the number of respondents rating impairment as 1 or higher. For those who reported “no” for serious difficulty hearing, 39.6% (n=36) reported a difficulty rating of 1 or higher. For those who reported “yes” 93.7% (n=22) reported a difficulty rating of 1 or higher. For those who reported “no” for serious difficulty seeing, 45.3% (n=48) reported a difficulty rating of 1 or higher. For those who reported “yes,” 88.9% (n=8) reported a difficulty rating of 1 or higher. For those who reported “no” for serious difficulty making decisions, 50.0% (n=43) reported a difficulty rating of 1 or higher. For those who reported “yes,” 92.9% (n=24) reported a difficulty rating of 1 or higher. For those who reported “no” for serious difficulty with stairs, 36.4% (n=20) reported a difficulty rating of 1 or higher. For those who reported “yes,” 96.6% (n=56) reported a difficulty rating of 1 or higher. For those who reported “no” for difficulty dressing or bathing, 31.5% (n=29) reported a difficulty rating of 1 or higher. For those who reported “yes,” 100.0% (n=21) reported a difficulty rating of 1 or higher. For those who reported “no” for difficulty running errands, 27.2% (n=75) reported a difficulty rating of 1 or higher. For those who reported “yes,” 93.9% (n=31) reported a difficulty rating of 1 or higher.

Note on Table 1: Total n varies due to variability in the number of respondents rating impairment as 1 or higher

 

This line graph shows the variability in difficulty scale rankings of those who answered “yes” to each impairment question. The x-axis shows the difficulty ranking scale from 0 to 10, and the y-axis shows percentage of respondents. There is little overlap between the lines representing each impairment, showing that there is greater variability in respondents’ difficulty scale rankings for each impairment.

Figure 1. Percentage of those who answered "yes" to the impairment question for each rating category

 

This line graph shows the lack of variability in difficulty scale rankings of those who answered “no” to each impairment question. The x-axis shows the difficulty ranking scale from 0 to 10, and the y-axis shows percentage of respondents. There is consistent overlap in the lines representing each impairment question, which shows the lack of variability in difficulty scale rankings. The majority of those who report no impairment, ranging from 50-72%, also report zero difficulty with the related ability or task.

Figure 2. Percentage of those who answered "no" to the impairment question for each rating category

 

Table 2. Reliability of dichotomous and scale impairment measures (n=119)

Table 2 shows the percentage of individuals rating an impairment difficulty as 1 or greater based on responses to the dichotomous measure. N=119. The table lists the percentage of respondents who indicated each category on the difficulty scale for those who responded “no” and “yes” for each impairment. • Impairment: Serious difficulty hearing o 79.0% (n=91) selected “no.” Of those who reported no:  60.4% (n=55) selected 0 on the difficulty scale  8.8% (n=8) selected 1  8.8% (n=8) selected 2  8.8% (n=8) selected 3  1.1% (n=1) selected 4  5.5% (n=5) selected 5  5.5% (n=5) selected 6  0% selected 7  0% selected 8  0% selected 9  1.1% (n=1) selected 10 o 21.0% (n=24) selected “yes.” Of those who reported yes:  8.3% (n=2) selected 0 on the difficulty scale  4.2% (n=1) selected 1  4.2% (n=1) selected 2  0 selected 3  8.3% (n=2) selected 4  16.7% (n=4) selected 5  8.3% (n=2) selected 6  25.0% (n=6) selected 7  12.5% (n=3) selected 8  4.2% (n=1) selected 9  8.3% (n=2) selected 10 • Impairment: Serious difficulty seeing o 92.4% (n=106) selected “no.” Of those who reported no:  54.7% (n=58) selected 0 on the difficulty scale  12.6% (n=13) selected 1  14.2% (n=15) selected 2  6.6% (n=7) selected 3  1.0% (n=1) selected 4  7.6% (n=8) selected 5  3.8% (n=4) selected 6  0% selected 7  0% selected 8  0% selected 9  0% selected 10 o 7.6% (n=9) selected “yes.” Of those who reported yes:  11.1% (n=1) selected 0 on the difficulty scale  0% selected 1  11.1% (n=1) selected 2  22.2% (n=2) selected 3  11.1% (n=1) selected 4  11.1% (n=1) selected 5  11.1% (n=1) selected 6  0% selected 7  11.1 (n=1)% selected 8  0% selected 9  11.1% (n=1) selected 10 • Impairment: Serious difficulty making decisions o 76.3% (n=86) selected “no.” Of those who reported no:  50.0% (n=43) selected 0 on the difficulty scale  22.1% (n=19) selected 1  15.1% (n=13) selected 2  5.8% (n=5) selected 3  2.3% (n=2) selected 4  1.2% (n=1) selected 5  1.2% (n=1) selected 6  0% selected 7  1.2% (n=1) selected 8  0% selected 9  0% selected 10 o 23.7% (n=28) selected “yes.” Of those who reported yes:  7.1% (n=2) selected 0 on the difficulty scale  3.6% (n=1) selected 1  7.1% (n=2) selected 2  3.6% (n=1) selected 3  10.7% (n=3) selected 4  14.3% (n=4) selected 5  7.1% (n=2) selected 6  21.4% (n=6) selected 7  17.9% (n=5) selected 8  3.6% (n=1)selected 9  3.6% (n=1)selected 10 • Impairment: Serious difficulty walking o 48.7% (n=55) selected “no.” Of those who reported no:  63.6% (n=35) selected 0 on the difficulty scale  7.3% (n=4) selected 1  7.3% (n=4) selected 2  5.5% (n=3) selected 3  0% selected 4  1.8% (n=1) selected 5  7.3% (n=4) selected 6  3.6% (n=2) selected 7  3.6% (n=2) selected 8  0% selected 9  0% selected 10 o 51.3% (n=58) selected “yes.” Of those who reported yes:  3.5% (n=2) selected 0 on the difficulty scale  3.5% (n=2) selected 1  6.9% (n=4) selected 2  0% selected 3  10.3% (n=6) selected 4  12.1% (n=7) selected 5  6.9% (n=4) selected 6  3.5% (n=2) selected 7  8.9% (n=10) selected 8  12.1% (n=7) selected 9  27.6% (n=16) selected 10 • Impairment: Difficulty dressing or bathing o 81.2% (n=92) selected “no.” Of those who reported no:  68.5% (n=63) selected 0 on the difficulty scale  8.7% (n=8) selected 1  6.5% (n=6) selected 2  8.7% (n=8) selected 3  1.1% (n=1) selected 4  2.2% (n=2) selected 5  1.1% (n=1) selected 6  2.2% (n=2) selected 7  1.1% (n=1) selected 8  0% selected 9  0% selected 10 o 18.8% (n=21) selected “yes.” Of those who reported yes:  0% selected 0 on the difficulty scale  0% selected 1  4.8% (n=1) selected 2  9.5% (n=2) selected 3  9.5% (n=2) selected 4  9.5% (n=2) selected 5  14.3% (n=3) selected 6  9.5% (n=2) selected 7  4.8% (n=1) selected 8  0% selected 9  38.1% (n=8) selected 10 • Impairment: difficulty running errands o 71.2% (n=81) selected “no.” Of those who reported no:  72.9% (n=59) selected 0 on the difficulty scale  12.4% (n=10) selected 1  3.7% (n=3) selected 2  3.7% (n=3) selected 3  2.5% (n=2) selected 4  1.2% (n=1) selected 5  1.2% (n=1) selected 6  0% selected 7  1.2% (n=1) selected 8  0% selected 9  1.2% (n=1)selected 10 o 28.8% (n=33) selected “yes.” Of those who reported yes:  6.1% (n=2) selected 0 on the difficulty scale  0% selected 1  9.1% (n=3) selected 2  9.1% (n=3) selected 3  6.1% (n=2) selected 4  6.1% (n=2) selected 5  6.1% (n=2) selected 6  9.1% (n=3) selected 7  9.1% (n=3) selected 8  6.1% (n=2) selected 9  33.3% (n=11) selected 10

Table 2 shows the percentage of individuals rating an impairment difficulty as 1 or greater based on responses to the dichotomous measure. For instance, 93.7% of those who answered “yes” to the “deaf or have serious difficulty hearing” question indicated that they some difficulty with hearing on the scaled item.  Those who answered “no” to the impairment questions sometimes indicated some difficulty using the scaled response option.

Discussion

Generally, when collected within the same time-period, forced-choice and scaled ratings of items in the six-question disability set demonstrate a high degree of concordance, especially for those who answer “yes.”  This suggests these items are reliable indicators of people experiencing some level of impairment.  Between 27.2% and 50.0% of individuals who do not report an impairment on the dichotomous items go on to rate some difficulty with the impairments.  This may be related to their interpretation of the word “serious” in four of the six items.

We have reported results that indicate many people change how they answer the dichotomous disability items (Ward et al.2017). The results presented here indicate that this change is not due solely to unreliability of the dichotomous items, but rather may reflect real change over time.  This change may reflect the dynamic nature of disability across environments or fluctuation in people’s impairment severity over time.  Future research using scaled responses will help to understand how impairment severity fluctuates over time.

References

Brault, M. W. (2009). Review of changes to the measurement of disability in the 2008 American Community Survey. US Census Bureau, 14. Retrieved from https://www.census.gov/people/disability/files/2008ACS_disability.pdf

Nagi, S. Z. (1979). The concept and measurement of disability. In E. D. Berkowitz (Ed.), Disability policies and government programs (pp. 1–15). New York: Praeger.

Siebers, T. (2008). Disability theory. Ann Arbor: University of Michigan Press.

Ward, B., Myers, A., Wong, J., & Ravesloot, C. (2017). Disability items from the current population survey (2008-2015) and permanent versus temporary disability status. American Journal of Public Health. Retrieved from http://ajph.aphapublications.org/doi/pdf/10.2105/AJPH.2017.303666

Verbrugge, L. M., & Jette, A. M. (1994). The disablement process. Social Science & Medicine, 38(1), 1–14. Retrieved from https://doi.org/10.1016/0277-9536(94)90294-1