Is It Biased? Empirical Analysis of Various Phenomena That Affect Survey Results

Abstract

It is often implied that survey results reflect only the quality of the sample and the underlying measuring instruments that were used in the survey. However, there are various phenomena that can affect the results whose influences are often neglected when conducting surveys. The aim of this study was to test the influences which various method effects have on survey results. We tested the influences of the following method effects: item wording, confirmatory bias, careless responding, and acquiescence bias. Using a split-ballot survey design with online questionnaires, we collected data from 791 participants. We tested if these method effects had an influence on mean values, item correlations, construct correlations, model fits, and construct measurement invariance. The instruments used to test these influences were from the domain of personality and gender inequality, and their items were adapted based on the method effect being tested. All tested method effects, with the exception of careless responding, had a statistically significant effect on at least one component of analysis. Item wording and confirmatory bias affected mean values, model fit, and measurement invariance. Controlling for acquiescence bias improved the fit of the model. This paper confirms that the tested method effects should be carefully considered when using surveys in research, and suggests some guidelines on how to do so.