Designing and Using Research Questionnaires

Designing and Using Research Questionnaires
Abstract
Purpose: This article draws on experience in supervising new researchers, and the advice
of other writers to offer novice researchers such as those engaged in study for a thesis, or
in another small-scale research project, a pragmatic introduction to designing and using
research questionnaires.
Design/methodology/approach: After a brief introduction, this article is organized into
three main sections: designing questionnaires, distributing questionnaires, and analysing
and presenting questionnaire data. Within these sections, ten questions often asked by
novice researchers are posed and answered.
Findings: This article is designed to give novice researchers advice and support to help
them to design good questionnaires, to maximise their response rate, and to undertake
appropriate data analysis.
Originality/value: Other research methods texts offer advice on questionnaire design and
use, but their advice is not specifically tailored to new researchers. They tend to offer
options, but provide limited guidance on making crucial decisions in questionnaire
design, distribution and data analysis and presentation.
Keywords: research questionnaires; quantitative research; quantitative data analysis.
Paper type: Conceptual paper
1

1. Introduction
Questionnaires are one of the most widely used means of collecting data, and therefore
many novice researchers in business and management and other areas of the social
sciences associate research with questionnaires. Given their prevalence, it is to easy to
assume that questionnaires are easy to design and use; this is not the case – a lot of effort
goes into creating a good questionnaire that collects the data that answers your research
questions and attracts a sufficient response rate. In this article, we use the term research
questionnaire to refer to questionnaires that are used as part of an academic research
project. Others (e.g. Bryman and Bell, 2011) use the term self-completion questionnaire,
or the related terms self-administered questionnaire or postal or mail questionnaire.
Further, we use the term questionnaire to refer to documents that include a series of open
and closed questions to which the respondent is invited to provide answers. Research
questionnaires may be distributed to the potential respondents by post, e-mail, as an
online questionnaire, or face-to-face by hand. Interviews, especially structured and semi-
structured interviews, also ask questions that the respondent is invited to answer, but the
essential distinguishing characteristic of questionnaires is that they are normally designed
to be completed without any direct interaction with the researcher, either in person or
remotely. However, the boundary between questionnaires and interviews is fuzzy, since
they are both question answering research instruments, with unstructured interviews at
one end of a spectrum and questionnaires comprised of predominantly closed questions at
the other end. Respondents to a questionnaire may be asked to answer questions
regarding facts (e.g. their age or salary), or their attitudes, beliefs, behaviours or
experiences as a citizen, manager, professional, user, consumer or employee. Since one
of the main advantages of questionnaires is the ability to make contact with and gather
responses from a relatively large number of people in scattered and possibly remote
locations, questionnaires are typically used in surveys, where the objective is to profile a
‘population’. This leads to consideration of who to include in the survey, or the sample.
In research in organizational studies, management, and business, participants may be
selected either as an individual or as a representative of their team, organization, or
industry.
2

If you are new to research, and possibly engaging in research to complete a thesis or
other small-scale project, and are planning to use questionnaires as a research method,
this article is written for you. It helps you to think about the decisions that you need to
make in designing questionnaires, distributing the questionnaires is such a way as to get a
good response rate, and analysing and presenting the data. This article seeks to provide
answers to some of the questions that new researchers frequently ask. Whilst its emphasis
is on helping you to do rigorous research and to succeed and maybe even excel, it is also
pragmatic in recognizing the time and other constraints often experienced by new
researchers.
There are many other sources of advice on designing and using research questionnaires
that you could also consult. First, there are many research methods textbooks that offer a
basic grounding in research methods (e.g. Bryman and Bell, 2011; Collis and Hussey,
2009; Cresswell, 2008; Denscombe, 2010; Easterby-Smith, Thorpe and Jackson, 2012;
Lee and Lings, 2008; Saunders, Thornhill and Lewis, 2012); since these books have a
wide scope, they only provide limited information on questionnaires as a data collection
method. Interestingly, there are only a few texts that deal specifically with quantitative
methods (e.g. Oakshott, 2009; Swift and Piff, 2010). Finally, there are a few texts
devoted specifically to questionnaires and/or surveys; amongst these Oppenheim (1992)
is regarded as a classic, whilst Gillham (2007), Sue and Pitter (2012) and Fowler (2008)
are also useful guides. Useful as these are, they can be a little daunting for the novice
researcher who is seeking a relatively quick and pragmatic approach to designing
questionnaires and analyzing their data. As with all research methods, learning how to
work with questionnaires is an iterative process, in which initial guidance allows the
researcher to get started, experience and reflection hones their art, and further advice
helps the researcher to develop their research skills yet further.
This article starts with discussion of a number of questions that are associated with the
design and planning of the questionnaire, and then moves on to consider aspects of the
questionnaire distribution and sampling, and finally, concludes with some thoughts on
making sense of the data and presenting it in a findings chapter.
3

2. Designing questionnaires
Q1. Why should I choose questionnaires for my research?
Questionnaires are mostly used in conducting quantitative research, where the researcher
wants to profile the sample in terms of numbers (e.g. the proportion of the sample in
different age groups) or to be able to count the frequency of occurrence of opinions,
attitudes, experiences, processes, behaviours, or predictions. For example, questionnaires
could be distributed to members of a social network site in order to ascertain the reasons
for their membership of the site, and the benefits that they perceive themselves to derive
from membership of the site. The questionnaire might include questions relating to any of
the standard topics included in questionnaires:
• ‘facts’, such as their age or occupation
• opinions, attitudes, beliefs and judgments, such as opinions on the benefits of the site,
attitudes towards various features or functions of the site, and perceptions of the
usability of the site
• behaviour, such as how frequently they visited the site.
Questionnaires are typically used in survey situations, where the purpose is to collect data
from a relatively large number of people (say between 100 and 1000). Often, but not
always, the people from whom responses are collected are a sample drawn from a wider
population, and are chosen to ‘represent’ the wider population. So, for example, if we
wanted to compare the leadership styles adopted by CEO’s in technology companies with
those of CEO’s in retail organizations in the UK, we are unlikely to be able to collect a
completed questionnaire from every CEO in these two sectors. So, we would need to
make a decision as to how many responses from CEO’s of what types of organizations
we would regard as sufficient, and select a sample accordingly.
Although there are many different approaches to collecting data with which
questionnaires can be compared, a common consideration for novice researchers is
whether to choose between questionnaires or interviews. The big advantage of
questionnaires is that it is easier to get responses from a large number of people, and the
4

data gathered may therefore be seen to generate findings that are more generalisable. For
example, if, say 400 students were surveyed on the factors that affected their choice of
mobile phone service provider, then this study would have the potential to be
generalisable to other members of the same student population. If, on the other hand,
instead of using questionnaires, the researcher had opted to conduct interviews, time
constraints would dictate that they collect data from rather fewer students, say, twenty.
With responses from only twenty students, we would feel a lot less confident that the data
collected would support generalization to the rest of the specific student population. On
the other hand, they may have potential to generate a range of insights and
understandings that might be useful, to say, mobile service providers. In general,
interviews are preferable to questionnaires when it is possible to identify people who are
in key positions to understand a situation, such as, say, the managers responsible for
implementing a corporate social responsibility policy in a specific brand of a retail chain.
In summary then, questionnaires are useful when:
• The research objectives centre on surveying and profiling a situation, to develop
overall patterns
• Sufficient is already known about the situation under study that it is possible to
formulate meaningful questions to include in the questionnaire.
• Willing respondents can be identified, who are in a position to provide
meaningful data about a topic. Questionnaires should not only suit the research
and the researcher, but also the respondents.
Q2. What types of research can be conducted through a questionnaire?
Surveys and questionnaires are employed to conduct a variety of different kinds of
research, key amongst which are:
1. Profiling and descriptive research, where the purpose is to generate a profile of
the characteristics of the sample. For example, in examining innovation in a
groups of SME’s in the food sector, questions might be posed to identify the level
of engagement with different innovation activities (Figure 1). Such research
5

answers questions such as what do they do, what do they think, and, what are their
characteristics (e.g organizational age).
2. Predictive and analytical research, where the purpose is to understand any
relationships between variables. We might be interested in the relationship
between the number of hours exercise that a manager takes a week and their BMI.
Provided we have asked the respondents for information on these two variables,
and we have a sufficiently large data set, we can look for patterns, using
techniques like correlation, regression, or chi-squared tests, to investigate the
relationship between these two variables. For example, does BMI go down as the
number of hours exercise a week goes up? More advanced techniques such as
multiple regression and structured equation modeling allow exploration of the
relationships between several variables at one time. Once research has established
relationships between variables it may be possible to offer some predictions as to
future events or patterns of behaviour.
3. Developing and testing measurement scales, where the purpose is to generate a
measurement scale, or a set of statements to measure a complex variable, such as
service quality, trust, or innovation orientation. Creating a measure such as the
number of years experience a person has in their current role, or the turnover for a
business in the previous financial year, is relatively straightforward. However,
measuring and hence asking questions that ‘measure’, for example, the
innovativeness of an organization, or the extent of formalization of its strategic
planning processes is much more complex. Accordingly, researchers develop
measurement scales, comprising of a number of statements that can be used to
measure the variable. Typically, they initially propose such statements based on
previous research, and then test and refine the scale using data collected from
appropriate respondents, with the aid of analytical methods such as exploratory,
principal components or confirmatory factor analysis. Only when they have such
measures of complex variables, can they ask questions such as ‘Is there any
6

relationship between the level of innovation orientation of the organization and its
turnover?’
Importantly, questions asked for one of above types of research can also be used in other
types.Thus, the responses to questions in Figure 1 can be used to determine either the
extent to which organizations are engaged in product innovation (profiling and
descriptive research), or they can be used together with questions on, say, organizational
age or turnover, to investigate the relationship between innovation activities and age and
turnover (predictive and analytical research). Finally, together with other similar
questions on innovation they can be used to generate an innovation orientation scale
(developing and testing measurement scales). In summary, it is important to be clear
about the aim and objectives of your research before embarking on questionnaire design.
Figure 1: Extract from a questionnaire on innovation orientation
1. How innovative are we?
Please respond on the following scale.
In our organisation we... Strongly
Agree Agree
Neither
agree or
disagree
Disagree Strongly
disagree
encourage new ideas throughout the
organisation.
encourage and support innovative
employees.
gather and use information about our
trade customers.
are effective at implementing change.
gather and use information about our
consumers/end-users.
put innovation at the heart of our
strategic planning.
gather and use information about our
competitors & markets.
engage in shaping an innovative
organisational culture.
7

Q3. How do I decide the questions to ask?
It goes without saying that the questions in the questionnaire are designed to generate
data that is intended to answer your research questions. On the other hand, the questions
often do not exactly match your research questions. First, and foremost it is important
that questions use language that respondents understand – whereas your research
questions may use more ‘academic’ or specialized technical language. Secondly, you
may for instance be interested in the relationship between two variables, such as in the
example above, the relationship between the manager’s BMI and their exercise regime.
The questionnaire is unlikely to ask this question directly (unless the aim is to explore
people’s opinions on the relationship). Rather, the questionnaire may ask about BMI and
exercise regime, separately, thereby collecting data that can be analysed to investigate the
relationship.
Both research and questionnaire questions can be informed by practice or experience, or
by theory or previous research, or, as is common with research in practitioner disciplines,
a mix of both. Research that is informed by previous theory and research is described as
deductive. With deductive research, theory is a significant factor in determining the
research questions, and indeed, it may be possible and even advisable to use part or all of
a previous questionnaire from a published article on a similar topic. Provided that you
acknowledge your sources, and the questions are adapted to your specific research
question, this is not cheating; you are using questions that have already been ‘piloted’ and
making it easier to compare your research with previous research and to make a clear
claim about what is new in your findings (Bryman and Bell, 2011). Indeed, there are
many instances in which replication, conducting a similar study to one that was
conducted earlier but in a different context, can be a valuable addition to knowledge.
Since the design of a questionnaire calls for some prior knowledge, deductive research is
more common in research using questionnaires, than the alternative approach, inductive
research, where the researcher deduces theory from the data that they have gathered.
When framing and designing your questions it is useful to think about the type of
question that is suitable for a specific context. The first, and most significant
8