Measuring impairment accurately is important for research and practice. Currently many researchers and practitioners depend on the six-item measure of impairment featured in the American Community Survey (ACS), which was defined and implemented in 2008 (see Brault, 2009 for a review of the development of this measure). Although this measure is widely used, responses to the items are not stable over time (Ward, Myers, Wong & Ravesloot, 2017). This study examined the reliability of these items when measured dichotomously or with a scaled response option.
In 2016, respondents of the 3rd wave of the Ecology of Rural Participation Survey were asked early in the survey to respond “yes” or “no” to the standard six questions measuring impairment regarding hearing, sight, cognitive abilities, physical abilities, performing daily tasks like bathing and dressing, and running errands. In an effort to better understand the range of difficulty experienced by people regarding impairment, later in the survey respondents were invited to rate how difficult these same conditions have been for them in the past 30 days using a 0-10 scale (i.e., 0 = no difficulty and 10 = extreme difficulty). We expected those who reported impairment on the forced-choice question to report some difficulty on the scaled items.
Respondents were very reliable in rating at least some difficulty with an impairment when they had answered “yes” to the forced-answer question (see Table 1). Generally, there is more variation in their reported levels of impairment when respondents endorsed an item compared to when they did not (see Figures 1 and 2).
Table 1. Percentage of respondents reporting “no” and “yes” to forced-choice question and more than zero on impairment scale
Note on Table 1: Total n varies due to variability in the number of respondents rating impairment as 1 or higher
Figure 1. Percentage of those who answered "yes" to the impairment question for each rating category
Figure 2. Percentage of those who answered "no" to the impairment question for each rating category
Table 2. Reliability of dichotomous and scale impairment measures (n=119)
Table 2 shows the percentage of individuals rating an impairment difficulty as 1 or greater based on responses to the dichotomous measure. For instance, 93.7% of those who answered “yes” to the “deaf or have serious difficulty hearing” question indicated that they some difficulty with hearing on the scaled item. Those who answered “no” to the impairment questions sometimes indicated some difficulty using the scaled response option.
Generally, when collected within the same time-period, forced-choice and scaled ratings of items in the six-question disability set demonstrate a high degree of concordance, especially for those who answer “yes.” This suggests these items are reliable indicators of people experiencing some level of impairment. Between 27.2% and 50.0% of individuals who do not report an impairment on the dichotomous items go on to rate some difficulty with the impairments. This may be related to their interpretation of the word “serious” in four of the six items.
We have reported results that indicate many people change how they answer the dichotomous disability items (Ward et al.2017). The results presented here indicate that this change is not due solely to unreliability of the dichotomous items, but rather may reflect real change over time. This change may reflect the dynamic nature of disability across environments or fluctuation in people’s impairment severity over time. Future research using scaled responses will help to understand how impairment severity fluctuates over time.
Brault, M. W. (2009). Review of changes to the measurement of disability in the 2008 American Community Survey. US Census Bureau, 14. Retrieved from https://www.census.gov/people/disability/files/2008ACS_disability.pdf
Nagi, S. Z. (1979). The concept and measurement of disability. In E. D. Berkowitz (Ed.), Disability policies and government programs (pp. 1–15). New York: Praeger.
Siebers, T. (2008). Disability theory. Ann Arbor: University of Michigan Press.
Ward, B., Myers, A., Wong, J., & Ravesloot, C. (2017). Disability items from the current population survey (2008-2015) and permanent versus temporary disability status. American Journal of Public Health. Retrieved from http://ajph.aphapublications.org/doi/pdf/10.2105/AJPH.2017.303666
Verbrugge, L. M., & Jette, A. M. (1994). The disablement process. Social Science & Medicine, 38(1), 1–14. Retrieved from https://doi.org/10.1016/0277-9536(94)90294-1