People have been sending out good surveys for a long time. However, they have been sending out bad surveys for about as long. Haven’t you ever wondered whether it would be possible to learn from these past successes and failures to help you design better surveys going forward?
That is basically what we’re working on here in the TrueSample team at SurveyMonkey. I have blogged previously about how to get better data quality by making sure that respondents are verifiable. How about getting better data quality by designing better surveys? No matter how honest or forthright a respondent is, we are all human. A badly designed survey is likely to make us lose concentration and, possibly, supply poor quality responses. To help design better surveys by learning from the past, we came up with SurveyScore.
In the past 3 years, TrueSample has processed over 50 million responses and tens of thousands of surveys, resulting in a vast database of survey responses. SurveyScore uses this database to create a score for your survey, ranging from 0 to 100, with 100 signifying a very good survey, and 0 signifying is a very bad one. A well designed survey with a high score is one in which respondents are likely to be highly engaged and provide good quality data. Conversely, a poorly designed survey with a low score is one where respondents are comparatively less engaged, and the quality of the data is also likely to be compromised.
SurveyScore is a combination of several metrics that we glean from surveys –
- Respondent rating of a survey (which is typically based on a scale of 1 – 6) that tells you whether their survey-taking experience was good – the greater this is, the more engaging the study.
- Respondent drop out rate (the % of respondents that do not complete the study after starting it) – the greater this % is, the less engaging the survey.
- The amount of speeding the respondents do during the study – the greater this is, the less engaging the study.
As you can see, SurveyScore can be considered to be quantitative representation of how engaging your study is.
Does survey design affect SurveyScore? All of this will be moot if we cannot show that survey design does not affect SurveyScore, right? If SurveyScore is independent of how well or poorly the survey is designed, then it is a meaningless metric. We set out to see if this was true.
In order to do so, we took over 1,600 studies from the past, with scores ranging from low to high, and extracted their design parameters. We extracted variables such as survey length, number of questions, number of words, number of pages, number of matrix (grid) questions, grid question complexity, number of consecutive grid questions etc. for each of these studies. Then we entered this into a machine learning algorithm, i.e. a predictive model, to see if the SurveyScore was predictive by this model. If the score is not predictive, then these design parameters will not affect how engaged respondents are in a study.
But lo and behold, we found that the score is actually quite predictive. We got an R-squared of close to 60.
This was very promising. But what are the main parameters that impact the score? They are what you would think they are – the longer the survey and more complex the study, the lower the score.
Since it is a multivariate problem, and there are a lot of interacting variables, it is not easy to change one parameter without affecting others when designing a survey. So we created a tool called SurveyScore predictor – it lets you input your design parameters and then launches the predictive model that tells you what your expected SurveyScore will be, and also tells you how to tweak your survey to improve your score.
So you can learn mathematically from the past to design better studies. We are continuing to work on improving the prediction by looking at other variables, model segmentation, etc to help you improve your studies, and get good quality data. In the end, we have created an objective, independent scoring system that summarizes past-learnings across tens of thousands of surveys to provide survey designers feedback on the impact their surveys on respondents (positive and negative). There is a lot going on under the hood of SurveyScore, but we help simplify the metric to a simple score between 0-100. Every TrueSample-certified survey gets a SurveyScore, so you know how your survey measures up against the legacy of surveys past.