The effect on the response rate of web surveys
How does the inclusion of the first question of the questionnaire in the survey invitation email in web surveys (CAWI) affect the response rate?
Low response rates in web surveys are a challenging problem. Researchers explore different response incentives, eg. when participants are invited by e-mail. However, we are only aware of two studies focused on questions embedded in the e-mail invitation (Varghese, Moore and Earnhart, 2018, Liu and Inchausti, 2017). The main purpose of this poster is to evaluate the impact of personalizing the invitation text by e-mail on the response. In particular, we evaluate the impact of an invitation by e-mail that includes the first question of the Web questionnaire with respect to a standard e-mail invitation on the completion of the questionnaire, the completion time, the interruptions and the composition of the interviewees.
Methods and data
We use the experimental data of a web survey conducted in the 2018 on the union delegates of the Italian General Confederation of Labor (CGIL).
The sample members (N = 5.494) were stratified by geographical area and type of trade union category, therefore they were randomly assigned (within the strata) to two groups: the group with the "link" and the group of e-mail invitation with the "first question" included. The e-mail text sent to the two groups was different only in the final statement. At the end of the e-mail text (the same for both groups), the first question of the questionnaire was reported in the "first question" group, while the link of the survey was included in the "link" group. The first question in the questionnaire is "How many years have you joined the Italian General Confederation of Labor union?" Delegate e-mails were not all available at the same time. For this reason the delegates received the invitation via e-mail (also called "first stimulus") at four different times, between 22 May and 15 June. To analyze our data we adopt both bi-varied and multi-varied analyzes.
Four main results stand out from our analyzes:
- The email invitation with the first embedded question is more effective in increasing the survey response than the invitation of the link only
- There are no differences in completion time
- The number of break-offs is higher in the "first question" group than in the "link" group
- There are no differences by age group, gender and CGIL registration
Completion of the questionnaires
From the bivariate analysis on the completion of the questionnaire we find that the overall response rate (calculated as the number of completed questionnaires divided by the number of e-mails sent to the initial sample of delegates, refers to RR1 defined by AAPOR - American Association for Public Opinion Research) is from 33%. Looking at the differences between the two experimental groups (1 graph), the response rate of the "first question" group is higher (37%) than that of the "link" group (30%).
To study the net effect of the experimental group on the propensity to response, we perform a binomial regression analysis with a progressive selection of the variables. As control variables we use the date of the first stimulus, the presence of the CGIL delegates in Italy (ie high, medium and small presence), the geographical area and the productive sector to which the delegates belong (not statistically significant). The results of the logistic regression model (1 Table) show that, ceteris paribus, the "first question" group has a higher probability of responding to the web survey than the control group (ie the "link" group). We specify that there are two main reasons for these results: i) the people who receive the e-mail invitation with the first question embedded in the text message, can immediately identify the e-mail as an invitation to the web survey and not as spam and email, and ii) the "first question" of an email invitation can arouse the curiosity of the interviewees.
As an additional response metric, in addition to the response rate, we calculated the interruptions, which we defined as individuals starting to fill out the questionnaire, but not sending it. Comparing the two experimental groups (3 graph), we found that the interruptions are higher for the "first demand" sample (16,8%) than for the "link" sample (12,5%). This difference is statistically significant (p value <1%).
To evaluate the net effect of the type of invitation by e-mail on break-offs, we perform a binomial regression analysis with the progressive selection of the variables (Table 3). After adjusting the effect of the invitation time (statistically significant), the geographical area (not statistically significant) and the presence of delegates in Italy (not statistically significant), we found a statistically significant difference between the group "first question" and the "link" group. In particular, respondents who received the invitation by embedded e-mail are more likely not to complete the survey.
We hypothesize that the invitation to embedded electronic mail is also effective in encouraging "reluctant" respondents to start the questionnaire. Even if some of these interviewees do not present the questionnaire (higher number of interruptions in the "first question" group compared to the "link" group), some others complete the survey, thus increasing the response rate for the "first" question "sample" .
Another indicator of the quality of the response we use is the time taken to complete the questionnaire. The 2 graph shows the distribution of completion time (for Web surveys completed within 45 minutes, the 91% of the sample) for each email invitation group.
The average completion time is 9,84 minutes for respondents who received the "first question" invitation, while it is from 9,61 minutes for those who received the "link" invitation.
We calculated a statistical test (ie t-test) to verify the differences between the two groups and we did not find any significant differences (value p = 38%). Looking at the gender and age range of respondents (Table 2), we did not find any differences in the completion time.
Composition of respondents
In this section we report the results on the composition of the respondents of each sample (ie the groups "first question" and "link"). We focused on three variables: gender, years of membership in the CGIL and age group.
When looking at gender, age group and registration in the CGIL, the composition of the sample is not influenced by the type of e-mail received.
We believe that our work can contribute to increasing the knowledge on the effectiveness of including a question in the e-mail invitation on the answer. In fact, as far as we know, this is the first study that examines the impact of the embedded invitation on completion times and the composition of respondents.
See the poster of the article.
Chiara Respi, University of Milan-Bicocca
Marco Fornea, Demetra Opinions.net Srl
Liu, M. and N. Inchausti. 2017.
Improving survey response rates: the effect of embedded questions in web survey email invitations.
10 Survey Practice (1)
Varghese, Moore and Earnhart, 2018;
Are Embedded Survey Items the Solution to Low Web Survey Response Rates? An Investigation of the Interaction Between Embedded Survey Items and Time of Survey Administration.
11 Survey Practice (2)