10.2 Collecting data using surveys

Data may be collected in many ways (laboratory experiments, taking measurements, field observations, etc.). For both observational and experimental studies, though, collecting data using surveys is common. Surveys are very difficult to do well: question wording is crucial, and surprisingly difficult to get right (Fink 1995).

Questions in a survey may be open-ended (respondents can write their own answers) or closed (respondents select from a small number of possible answers, as in multiple-choice questions). Both open and closed questions have advantages and disadvantages. Answers to open questions usually lend themselves to qualitative analysis.

This section only briefly examines surveys:

• Asking survey questions; and
• Comparing online and paper surveys.

10.2.1 Asking survey questions

Many issues must be kept in mind when framing survey questions; here are some.

• Avoid leading questions which may indicate how respondents are expected to answer. Question wording is the usual reason for leading questions.

• Avoid ambiguity: Avoid terms that may be unfamiliar, and questions that are unclear.

• Avoid asking the uninformed, and avoid asking respondents about issues they don’t know about (for example, people will even give directions to places that do not even exist (Collett and O’Shea 1976)). Many people will tend to give a response even if they do not understand, but such responses are worthless.

• Avoid complex and double-barrelled questions; these are often hard to understand.

• Avoid problems with confidentiality, which would be considered unethical. Ethics committees usually look very carefully for question that are unethical. In special cases and with justification, ethics committees may allow such questions.

Example 10.3 (Leading question) This survey question is a leading question, because the expected response is obvious:

Because bottles from bottled water create enormous amounts of non-biodegradable landfill and hence pose a threat to sensitive native wildlife, do you support a ban on bottled water at USC?

Example 10.4 (Question wording) Question wording can be important. These two questions would produce different percentages of respondents agreeing:

• Which is easier to buy: cigarettes, beer or marijuana?
• Which is easier to obtain: cigarettes, beer or marijuana?

Example 10.5 (Leading question) Consider this survey question:

Do you like this new orthotic?

Although not obvious, this question may incite respondents to please, since liking is the only option presented. Better would be to ask:

Do you like or dislike this new orthotic?

Even better (but more difficult to implement) is to ask the second question above, but randomly chose the order of the ‘like’ and ‘dislike’; that is, ask some respondents if they ‘like or dislike’ the new orthotic, and others if they ‘dislike or like’ the new orthotic.

Example 10.6 (Ambiguous question) Consider this survey question:

Do children run faster now?

This question is ambiguous: Faster now compared to what or when?

Example 10.7 (Asking the uninformed) Consider this survey question:

Is the use of fibre composites for waterside recreational purposes likely to cause the material to swell?

Only people involved in the industry are likely to be able to properly answer this question. Nonethless, many people will still give an opinion, even if they are uninformed. This data will be effectively useless (response bias), but the researcher may not realise this.

Example 10.8 (Unclear wording) Consider this survey question:

I don’t go out of my way to purchase low-fat food unless they are also low in calories but not necessarily salt. Do you agree?

It is not clear what a ‘yes’ answer means.

Example 10.9 (Double-barrelled question) Consider this survey question:

Do you jog and swim for exercise?

This question would be better asked as two separate questions: one asking about jogging, and one about swimming.

Example 10.10 (Confidentiality) Consider this survey question:

Do you have a water tank that has been installed illegally, without council permission?

Respondents are unlikely to admit to breaking rules.

Think 10.1 (Survey questions) Consider this survey question:

Consider this book that you are currently reading. How useful do you think this book would be for students and young professionals in the field?

What is the biggest problem with this survey question?

There are two questions; it is double-barrelled. Better is to ask the two questions separately, one about students and one about young professionals. This allows us to separate the two components of the original survey question.

The following (humourous) video shows how survey questions can be manipulated by those not wanting to be ethical:

10.2.2 Online and paper surveys

Surveys may be conducted using paper-based surveys, or online surveys; both have advantages and disadvantages (Porter 2004).

Paper-based surveys require the survey information to be manually entered into the computer for later analysis, which is time consuming and expensive, and prone to data-entry errors. Paper-based surveys can also be costly to prepare, especially if physical mailing and photocopying is necessary. However, people may be more likely to complete paper-based surveys if they are presented with a survey face-to-face and someone waits to collect the completed survey.

Online surveys make data collection easier and data entry easier: data are entered directly onto a computer. This means less manual handling and less chance of data entry errors. Online surveys are also easier to share with a geographically-diverse group of people (for example, through email or social media), but only if the relevant contact details are available. However, online surveys may have a lower response rate, as respondents may be reluctant to click on links in emails (especially from unknown sources), may ignore emails, or the emails may be flagged a spam.