The article of which we offer you a summary investigates the phenomenon of misreporting motivated in a web panel. The misreporting motivated is the mechanism by which respondents to a survey provide incorrect or mismatched answers to their opinions for the sole purpose of reducing the length of the questionnaire and therefore the burden in terms of time and energy necessary for its completion.
Many surveys use filter questions to determine the eligibility of question respondents follow-up. Their aim is to reduce the burden on respondents by asking only relevant questions. Two common formats are used for these filter questions:
1. In format interleafed respondents are asked a filter question and questions follow-up, if activated, follow immediately;
2. In format grouped respondents are first asked all the filter questions and then the questions follow-up;
Several studies have shown that respondents have access to fewer follow-up with a format interleafed that with a grouped format (Eckman, Kreuter, Kirchner, Jackle, Tourangeau, et al., 2014). This happens because in the format interleafed respondents learn how the survey is structured and start saying “No” to the filter questions to avoid the questions follow-up, while this possibility is reduced in the bundled format in which the filter questions are placed all together.
A similar mechanism underlies a form of conditioning for panel surveys.
In fact, the interviewees can learn from the first ones wawe how the questionnaire is structured and use this information to respond incorrectly in the wawe later.
Halpern-Manners and Warren (2012) have found confirmation of this hypothesis in the Current Population Survey.
A systematic review
In the study, three hypotheses were tested.
1. Iprobing conditioning potesi: the number of questions by follow-up activated depends on the format with which they are placed (interleafed or grouped)?
2. assumptions conditioning of the panel: misreporting motivated remains in the different wawe?
3. Hypothesis of panel conditioning in modified format: misreporting motivated varies when respondents are interviewed in different formats in each wawe (ie, when the format changes from grouped to interleAFED and viceversa)?
In addition to testing these three hypotheses, it has been verified whether the conditioning of the survey and the conditioning of the panel depend on the cognitive abilities of the interviewees.
The data of the study come from the Dutch panel LISS, an Internet panel used for many years and based on a probabilistic sample. Sample members fill out questionnaires of around 15-30 minutes on a monthly basis (Scherpenzeel and Das 2010).
The results
As predicted by the survey conditioning hypothesis, the percentage of filter questions activated in the first wawe it is significantly smaller in size interleafed with respect to the grouped format: 36% against 43% (ie 4.7 against 5.6 activated filters). The conditioning of the survey takes place in the first wawe as expected. Another significant aspect is that the probability of activating a filter in the format interleAFED decreases significantly as we progress through the various sections of the questionnaire. This finding further supports l'Ithat the respondents could do learn to use the misreporting in the format interleafed. Regarding the second hypothesis concerning the conditioning of the panel, respondents interviewed in format interleafed in both phases they do not show significant changes over time in relation to the answers provided and the same applies to respondents interviewed in the format grouped in both wawe (third hypothesis). Therefore no support is found for the hypothesis of panel conditioning and for the hypothesis of panel conditioning in modified format.
Regarding the influence of cognitive abilities, there is no data to support the fact that the filters activated in the two formats vary according to the level of education of the interviewees.
In summary: the results show experimental evidence in favor of the survey conditioning hypothesis: nthe format interleafedfewer filters are activated compared to the grouped format, and there is no evidence that respondents remember survey structure over time in a panel survey context and in the misreporting over the second wawe. Therefore, there is no evidence to support the hypothesis regarding the mechanism of worsening of respondents over time.
Conclusions
Survey professionals are increasingly aware that asking questions about filters in different formats can affect measurement responses and errors. Studies on the misreporting motivated in the transversal investigations showed that the questions asked in a format interleafedhave activated fewer follow-up compared to respondents interviewed in a grouped format. The common explanation for this conditioning is that respondents learn how a survey is structured and use this information to shorten the questionnaire and reduce its burden.
Basically, while respondents show a strong tendency to remember the structure of a questionnaire in a single survey, they do not seem to recall this information when the same questions are repeated over time. Also respondents in format interleafed in the first wawe, many of whom misreporter, no longer show any sign of misreportingwhen they are interviewed for the second time in a grouped format.
The most likely explanation for this finding is the reset effect reported by Kreuter et al. (2011) They argue that the misreporting of respondents are reset when a new section begins within a questionnaire. Similar logic could apply to a panel survey context: if the erroneous reporting is reset when starting a new section, the same should apply for starting a new one. wawe in a panel survey.
Although the misreporting can be a serious problem, repeated participation in the same survey does not make the problem worse while the use of a grouped format is preferable to the format interleafed.
Our ariculture reworking: Motivated Misreporting in Web Panels
by Ruben L. Bach and Stephanie Eckman for AAPOR
