Selection Bias and Representation of Research Samples

The effectiveness of Mixing Mode and Sampling Frames

Research background and questions

Nowadays mixed-mode approaches are used to address the problem of non-coverage and non-response error in sample surveys. In the literature there are many examples of surveys combining web, telephone and F2F modalities, adopting concurrent or sequential designs in experimental or non-experimental studies.

The issues of interest are varied, eg experiments on the format of questions, differences in terms of response and coverage, bias on social desirability and estimates of data quality. Furthermore, the studies often use the same sampling scheme for mixed-mode detection and show conflicting results on the differences between the samples.

In this context, drawing on our previous work (Bartoli, Respi and Fornea, 2018), we apply a survey design in mixed mode to different sampling frames (list of landline phones and online panel). The problem with phone coverage is compounded by the fact that families with landlines are not equally represented in the entire Italian population. We hypothesize that this source of bias, combined with the no response error, could be reduced by adopting approaches using different sample frames.

This research aims to evaluate the representativeness of the samples of a survey design in mixed mode (web-landline) and a telephone survey that calls landline and mobile telephone numbers, comparing their estimates with the values ​​observed from the registered voter registers and with the socio-economic characteristics of the Italian population.

Method

To evaluate the representativeness of the samples, we first compare the voting behavior estimated by the two survey designs to the voting behavior observed (“true” values) in the last political elections.
We perform bivariate analyzes and use the mean absolute error, the largest absolute error and the differences in percentage points as accuracy metrics.

In addition, we also compare the employment status and training of our respondents with those of the respondents in the Labor Force Survey, calculating (as an accuracy metric) the percentage error for the modal category of the benchmark.

Data

We use data from 6 telephone and web surveys conducted in Italy (period March 2018 - January 2019) on landline phones or mobile phone owners and on members of Opinione.net, Italian online panel.

We designed a sequential mixed-mode survey (a computer-assisted web interview - CAWI survey followed by a computer-assisted telephone interview - CATI survey, using two different sample frames) and a survey with two different sample frames (computer-assisted mobile phone interview - CAMI survey followed by a CATI survey).

The same research institute commissioned all the investigations to Demetra opinione.net Srl., which divided each sample (about 1.000 panelists / respondents) equally between the two survey projects: half of the respondents compared to the mixed mode design and half of the telephone survey.

Sampling

Both projects adopt a sampling of quotas: quotas were defined proportional to gender, age and geographical area of ​​the distribution of residence of the Italian population (we used administrative data from the data.istat site as a reference point) . The questionnaires used for each poll are different, but cover all political attitudes and behavior, and include a question with the same word question about voting in the last national elections (i.e. "In the national elections of March 4, 2018, which party votes for? ”), which we used in our analyzes.

In our previous work (Bartoli, Respi e Fornea, 2018) we used a question about the vote taken in 2014, while in this study we focus on the vote in 2018. Our investigations were conducted in the months immediately following the general election. It is assumed that the "memory effect" did not occur, thus removing a potential source of bias from the estimates of voting behavior.

1 table. Overview of the two study designs

2 table. Survey overview

We also use secondary data as a benchmark:
the database of registered voters (2018),
and the Labor Force Survey (2017).

Results: representativeness of voting behavior

The results of the analyzes on the extent of bias in the estimates of voting behavior show some differences between the CAMI - CATI and CAWI - CATI models.

In particular, Table 3 focuses on the mean absolute error and shows that the CAWI - CATI design outperforms CAMI - CATI in all six surveys, when representing the voting behavior.

3 table. Average absolute error for the question on the voting behavior of each survey sample

Thus, we focus on the category of the demand on voting behavior with the biggest absolute error for each survey sample. The 4 Table reports the results.

4 table. Maximum absolute error for each survey sample.

• Democratic Party

• Forza Italy

Three main results stand out:

1. the values ​​of the largest absolute error are lower for CAWI-CATI than for the CAMI-CATI survey model;
2. CAMI-CATI champions tend to systematically overestimate the people who vote "Democratic Party";
3. CAWI-CATI samples tend to systematically represent the people who vote for “Forza Italia”.

Finally, looking at the percentage differences for the main parts (1-4 graphs), we find some models.

Graph 1. Differences in percentage points for the 'Democratic Party'.

The "Democratic Party" is always over-represented. The differences are systematically higher for CAMI-CATI than for the CAWI-CATI survey model, with the exception of the “September 2018” survey in which the CAMI - CATI sample performs better than CAWI-CATI. The trend of the telephone survey shows greater variability than the series of mixed-mode surveys.

Graph 2. Differences in percentage points for “Forza Italia”.

The 'Forza Italia' party is always under-represented in both survey designs. Four out of six CAWI-CATI samples have a higher bias than telephone ones.

Graph 3. Differences in percentage points for “Lega Nord”.

The “Lega Nord” party shows an interesting pattern for both polling projects: “Lega Nord” voters are under-represented in the first three polls, while, since September 2018, they have always been over-represented. Overall, the percentage differences increased systematically for both survey projects from March 2018 to January 2019.

Graph 4. Differences between percentage points for “5 Star Movement”.

The 'Movimento 5 Stelle' party does not show a clear pattern during the polls. We can say that the CAWI-CATI respondents are more likely to be the voters of "Movimento 5 Stelle" (except for the survey conducted in September 2018) than the CAMI-CATI respondents. However, the most recent survey (i.e. January 2019) reports very low (and equal) percentage point differences for both mixed-mode and telephone samples.

Results: representation of socio-demographic characteristics

The analysis of the distortions in the estimates of the socio-economic characteristics of our survey samples shows high values ​​for both the CAWI-CATI and CAMI-CATI survey designs. Graphs 5 and 6 show the percentage point error for the modal categories of employment status (i.e. inactive persons) and education (i.e. lower secondary education). Graph 5 shows the percentage of CAMI-CATI versus CAWI-CATI.

5 chart. Percent error for inactive people.

CAMI-CATI samples systematically underrepresent inactive people (graph 5). The CAWI-CATI survey design offers better performance in the representation of this category. In fact, compared to the telephone series, the extent of the error is always lower, with the exception of the last survey carried out in January 2019.

6 chart. Percent error for lower secondary education.

Figure 6 shows that both telephone and mixed-mode survey samples are not representative of people who have not passed lower secondary education: all samples underrepresent this category and the differences with the general population are very marked (at least -18 percentage points).

Conclusions

We compared the estimates of six telephone surveys and in mixed mode with reference data and we evaluated the representativeness of the sample. We focused our analysis on voting behavior, employment status and training of respondents. The results are consistent with those of our previous work (Bartoli, Respi and Fornea, 2018) and show that the mixing of both modes and the sampling frame, as in the CAWI-CATI survey, is a more effective strategy in reducing errors of selection. In particular, the following main results are highlighted.

1. The design of the CAWI-CATI survey performs better than the CAMI-CATI survey in representing the overall voting behavior:
   • a) the average absolute error value is lower in all surveys;
   • b) the magnitude of the largest absolute error is smaller in all surveys.
2. Looking at the parts selected by the interviewees, we identified four main models:
   • a) CAMI-CATI champions tend to systematically overestimate the people who vote "democratic party";
   • b) CAMI-CATI champions underrepresent people who vote for “Movimento 5 Stelle”;
   • c) both the CAMI-CATI and CAWI-CATI champions underrepresent the people who vote for “Forza Italia”;
   • d) there are no differences between the two survey designs when depicting people voting for "Northern League", but all the differences have increased over time.
3. The design of the CAWI-CATI survey performs better than the CAMI-CATI one in representing the employment status of the Italian population. In fact, the magnitude of the percentage error for inactive people is always smaller, except in the latest survey.
4. Both the CAWI-CATI and CAMI-CATI survey designs end up not representing people who have not passed lower secondary education.

Also read the survey poster: Poster
Authors:
Dr. Chiara Respi, University of Milan-Bicocca, Italy
Dr. Beatrice Bartoli, Dr Marco Fornea, Demetra Opinions.net Srl