Probably not probabilistic, this is not the problem!

In this special issue of POQ's 2017 (Public Opinion Quarterrly) the discussion on the quality of the investigations continues.

In the introduction to this volume, the question is asked about the effectiveness of surveys with Response Rate (RR) below 10%. This question is now recurring in many publications. Furthermore, the increasingly massive use of non-probabilistic surveys on online panels is posing the problem of the effectiveness of this methodology.

This article addresses the problem in a pragmatic way by trying to quantify the bias of samplings non-probabilistic online panels and samplings probabilistic in surveys with low response rates. It also tests the ability of the various weighting techniques to correct the bias.

Noteworthy is the method:

  • compares the crossings of the socio-demographic variables common to 4 non-probabilistic surveys and 3 probabilistic surveys with a 10% lower RR with the crossings of the same variables in a probabilistic survey with a higher than 80% RR;
  • applies 3 different weighting techniques (raking, propensity and matching) to verify the ability of these techniques to reduce the bias.

The conclusions are

  • non-probabilistic samplings sometimes have a double bias of probabilistic investigations with a low Response Rate;
  • weight techniques fail to reduce the bias systematically in non-probabilistic investigations.

These conclusions, if associated with the cost of the two sampling techniques can have considerable practical consequences.

Apples to oranges or gala versus golden delicious?
Comparing data quality of nonprobability internet samples to low response rate probability samples

by david dutwin trent d. Buskirk

Leave a comment