The effect on the response rate of web surveys
How does the insertion of the first question in the survey in the survey invitation email in web surveys (CAWI) affect the response rate?
Low response rates in web surveys are a challenging problem. Researchers explore different response incentives, e.g. when inviting attendees by email. However, we are only aware of two studies focusing on questions incorporated into the email invitation (Varghese, Moore and Earnhart, 2018, Liu and Inchausti, 2017). The main purpose of this poster is to assess the impact of personalizing the email invitation text on the response. Specifically, we evaluate the impact of an email invitation that includes the first question of the web questionnaire versus a standard email invitation on the completion of the questionnaire, time to complete, interruptions and composition of respondents.
Methods and data
We use the experimental data of a web survey conducted in the 2018 on the union delegates of the Italian General Confederation of Labor (CGIL).
The members of the sample (N = 5.494) were stratified by geographical area and type of trade union category, then they were randomly assigned (within the strata) to two groups: the group with the "link" and the group of invitation to email with the “first question” included. The text of the email sent to the two groups was different only in the final statement. At the end of the e-mail text (the same for both groups), the first question of the questionnaire was reported in the "first question" group, while the survey link was included in the "link" group. The first question of the questionnaire is "How many years have you joined the Italian General Confederation of Labor union?". Delegate emails were not all available at the same time. For this reason the delegates received the invitation via e-mail (also called “first stimulus”) at four different times, between 22 May and 15 June. To analyze our data we use both bi-varied and multi-varied analyzes.
Four main results stand out from our analyzes:
- The email invitation with the first question embedded is more effective in increasing survey response than just the link invitation
- There are no differences in completion time
- The number of break-offs is higher in the "first question" group than in the "link" group
- There are no differences by age group, gender and CGIL registration
Completion of the questionnaires
From the bivariate analysis on the completion of the questionnaire we find that the overall response rate (calculated as the number of completed questionnaires divided by the number of e-mails sent to the initial sample of delegates, refers to RR1 defined by AAPOR - American Association for Public Opinion Research) is 33%. Observing the differences between the two experimental groups (graph 1), the response rate of the "first question" group is higher (37%) than that of the "link" group (30%).
To study the net effect of the experimental group on response propensity, we perform a binomial regression analysis with a progressive selection of variables. As control variables we use the date of the first stimulus, the presence of the CGIL delegates in Italy (i.e. high, medium and small presence), the geographical area and the production sector to which the delegates belong (not statistically significant). The results of the logistic regression model (Table 1) show that, ceteris paribus, the "first question" group has a higher probability of responding to the web survey than the control group (ie the "link" group). We specify that there are two main reasons for these results: i) People who receive the email invitation with the first question embedded in the text message, can immediately identify the email as a web survey invitation and not such as spam e-mail, and ii) the “first question” of invitation via e-mail can arouse the curiosity of the interviewees.
As an additional response metric, in addition to the response rate, we calculated outages, which we defined as individuals who begin filling out the questionnaire, but not submit it. Comparing the two experimental groups (graph 3), we found that the interruptions were higher for the “first question” sample (16,8%) compared to the “link” sample (12,5%). This difference is statistically significant (p-value <1%).
To evaluate the net effect of the email invitation type on break-offs, we perform binomial regression analysis with progressive selection of variables (Table 3). After adjusting the invitation time effect (statistically significant), the geographical area (not statistically significant) and the presence of delegates in Italy (not statistically significant), we found a statistically significant difference between the “first question” group and the “link” group. In particular, respondents who received the embedded email invitation are more likely not to complete the survey.
We hypothesize that the embedded email invitation is also effective in encouraging "reluctant" respondents to start the questionnaire. Although some of these respondents do not submit the questionnaire (higher number of interruptions in the "first question" group than in the "link" group), some others complete the survey, thus increasing the response rate for the "first" question "sample" .
Another indicator of the quality of the response we use is the time taken to complete the questionnaire. The 2 graph shows the distribution of completion time (for Web surveys completed within 45 minutes, the 91% of the sample) for each email invitation group.
The average completion time is 9,84 minutes for respondents who received the “first question” invitation, while it is 9,61 minutes for those who received the “link” invitation.
We calculated a statistical test (ie t-test) to verify the differences between the two groups and we did not find any significant differences (value p = 38%). Looking at the gender and age range of respondents (Table 2), we did not find any differences in the completion time.
Composition of respondents
In this section we report the results on the composition of the interviewees of each sample (ie the “first question” and “link” groups). We focused on three variables: gender, years of enrollment in the CGIL and age range.
When looking at gender, age range and CGIL membership, the composition of the sample is not affected by the type of email received.
We believe our work can help expand knowledge about the effectiveness of including a question in the email invitation about the answer. In fact, to the best of our knowledge, this is the first study examining the impact of the embedded invitation on completion times and the composition of respondents.
See the poster of the article.
Chiara Respi, University of Milan-Bicocca
Marco Fornea, Demetra Opinions.net Srl
Liu, M. and N. Inchausti. 2017.
Improving survey response rates: the effect of embedded questions in web survey email invitations.
10 Survey Practice (1)
Varghese, Moore and Earnhart, 2018;
Are Embedded Survey Items the Solution to Low Web Survey Response Rates? An Investigation of the Interaction Between Embedded Survey Items and Time of Survey Administration.
11 Survey Practice (2)