To investigate the advantage of mixed-mode (MM) surveys, it is necessary to evaluate selection effects between modes. The selection effects refer to the differences in the compositions that correspond to the target variables between the modalities.
However, estimating selection effects is not an easy task because they can be completely confused with measurement effects between modes (differences in measurement error). Publications concerning the estimation of these mode effects are scarce.
The European social survey (ESS) began in the 2002 as a two-year survey on how to change social values and attitudes in Europe.
In order to encourage equivalence between different countries, the main investigations of all waves have been conducted until now mainly by face-to-face personal interviews. However, due to the cost of this type of interviews and the decrease in response rates, small mixed-mode surveys were established in parallel to the main single-mode surveys. This could result in a minor selection error.
With an investigation mixed-mode, you can use larger samples with the same budget limits. In this case, the mixed mode survey can offer greater external reliability than the single mode survey.
However, data collection methods can affect quality in particular.
A measurement effect occurs if two data collection modes provide different quality data for the same group of respondents. Measurement effects are problematic because, in the first place, they threaten the validity of comparisons between respondents of different data collection modalities and, secondly, they also threaten the comparability of data collected in mixed mode with other single mode data. It is difficult to analyze the effects on data quality because measurement effects can be confused with selection effects. Selection effects occur when groups of respondents selected for different modes differ on target variables. This document uses the instrumental variable method as a method for examining the measurement and selection effects on the quality of ESP data estimates obtained from the Multitrait-Multimethod (MTMM) experiments.
Methods and Data:
The analysis data of this article derive from a mixed mode survey that was created in parallel to the fourth wave of the main European social survey (ESS) in the Netherlands in the 2008-2009. In mixed mode design, sample members were asked to respond through one of three ways of collecting data, namely:
- a self administered web questionnaire (WSAQ)
- a computer-assisted telephone interview (CATI)
- a CAPI
Both the main survey and the mixed mode survey started with two independent random samples. The mixed mode survey started with a sample of 1756 people with a matching phone number while the main survey started with a sample of 2674 people with a matching phone number.
To correct differences in family size, standardized design weights that are proportional to the size of the family are used in all analyzes.
Both the main ESS and mixed mode samples are weighed separately on a set of socio-demographic variables.
Each respondent can theoretically be represented by two rows of data instead of one, where each represents the respondent's response when a specific data collection mode was used.
We define the variable D that refers to the data collection mode and accepts both values:
"P" refers to the lines in which the data of the respondents were collected by CAPI
"Tw" refers to the data lines where they were collected by CATI or WSAQ depending on the choice made.
We also define the variable G, which refers to the group of modalities in which a respondent has been really selected in the mixed-mode survey. Like D, this variable takes on value p if CAPI, tw if CATI-WSAQ.
The Y target variables, used to calculate the quality of the data, can concern both D and G.
By definition, Y is causally influenced by the data collection mode D because the mode defines the response measurement error. The relationship between G and Y instead reflects a selection effect, as it implies differences in the composition of respondents among the modalities.
In the ideal situation, the answers of all the interviewees are observed in both the tw mode, so there is no relationship between D and G, as two rows of data can be theoretically defined for each respondent, one for each collection mode data, regardless of the actual mode group for which the respondent was selected in the mixed-mode survey.
Of course, in practice some of these data lines are not observed and this can cause problems in estimating selection and measurement effects.
In fact, within the observed data, the selected G mode group completely determines the compilation mode in D for each respondent.
An instrumental variable (I) interrupts the relationship between the variables G and D, in this way the measurement and selection effects become partially distinguishable.
Risultati:
The overall effects of the modalities are quite small, and the most extreme effect is on the question of trust in the Dutch parliament. In general, the effects of the modality are caused by the differences in the composition of the respondents among the different modalities. It can be concluded that data collection methods generally do not have a major effect on their quality, but particular attention should be given to some questions and scales of response. This conclusion is an argument in favor of the further use of mixed mode surveys. It should be noted, however, that the instrumental variable makes it possible to estimate the effects of measurement and selection only if both the hypothesis of equivalence of measurement between samples and the assumption of representativeness are present. These hypotheses, unfortunately, due to design deficiencies such as the small size of the sample and the particular type of population with which you are dealing (the Netherlands with a high Internet coverage) can not be fully validated and are required further future studies.
Authors:
Jorre TA Vannieuwenhuyze, Melanie Revilla
