The description of the data gathering instruments, such as questionnaires, interview schedules, tests, inventories, etc., should be adequately detailed to provide the source of the instrument, any special characteristics (including its validity and reliability), any special restrictions or limitations, and its most appropriate use. Standardized instruments usually have established validity and reliability either from a history of use or from actual tests, while researcher-designed instruments usually do not have established validity and reliability. They may need to be validated by a panel of experts and should be pilot-tested where possible.

Questionnaire Design

When one looks at a completed questionnaire or the results of a valid study, he or she is often tempted to say, “That’s not difficult to do”, but that temptation quickly passes after a few minutes of closer analysis. That’s about all it takes to realize that designing a good question and good questionnaire requires more thought and time than one might originally think. There are an almost infinite variety of things to think about and to do correctly to avoid the kinds of errors that can make scores of hours of work worthless. A good questionnaire is one which helps directly achieve the research objectives, provides complete and accurate information; is easy for both interviewers and respondents to complete, is so designed as to make sound analysis and interpretation possible and is brief. Hence, questionnaire design is more of an art than a science. There are nine steps involved in the development of a questionnaire.

1. Decide the information required

It should be noted that the first step is to decide “what are the things one needs to know from the respondent in order to meet the survey’s objectives?” These should appear in the research brief and the research proposal. One may already have an idea about the kind of information to be collected, but additional help can be obtained from secondary data, so to speak, the researcher should be aware of what work has been done on the same or similar problems in the past, what factors have not yet been examined, and how the present survey questionnaire can build on what has already been discovered.

2. Define the target respondents

At the outset, the researcher must define the population about which he/she wishes to generalize from the sample data to be collected. Secondly, researchers have to draw up a sampling frame. Thirdly, in designing the questionnaire we must take into account factors such as the age, education, etc. of the target respondents.

3. Choose the method(s) of reaching your target respondents

The method of contact will influence not only the questions the researcher is able to ask but the phrasing of those questions. The main methods available in survey research are personal interviews; group or focus interviews; mailed questionnaires; telephone interviews. A general rule is that the more sensitive or personal the information, the more personal the form of data collection should be.

4. Decide on question content

Researchers must always be prepared to ask, “Is this question really needed?” The temptation to include questions without critically evaluating their contribution towards the achievement of the research objectives, as they are specified in the research proposal, is surprisingly strong. No question should be included unless the data it gives rise to is directly of use in testing one or more of the hypotheses established during the research design.

5. Develop the question wording

Survey questions can be classified into three forms, i.e. closed, open-ended and open response-option questions. Closed questions have a number of important advantages:

(a) It provides the respondent with an easy method of indicating his answer—he does not have to think about how to articulate his answer.

(b) It “prompts” the respondent so that the respondent has to rely less on memory in answering a question.

(c) Responses can be easily classified, making analysis very straightforward.

6. Put questions into a meaningful order and format

Opening questions: Opening questions should be easy to answer and not in any way threatening to the respondents. The first question is crucial because it is the respondent’s first exposure to the interview and sets the tone for the nature of the task to be performed. If they find the opening question easy and pleasant to answer, they are encouraged to continue.

Question flow: Questions should flow in some kind of psychological order, so that one leads easily and naturally to the next. Questions on one subject, or one particular aspect of a subject, should be grouped together. Respondents may feel it disconcerting to keep shifting from one topic to another, or to be asked to return to some subject they thought they gave their opinions about earlier.

Question variety: Respondents become bored quickly and restless when asked similar questions for half an hour or so. It usually improves response, therefore, to vary the respondent’s task from time to time.

7. Check the length of the questionnaire

In general it is best for a questionnaire to be as short as possible. A long questionnaire leads to a long interview and this is open to the dangers of boredom on the part of the respondent (and poorly considered, hurried answers), interruptions by third parties and greater costs in terms of interviewing time and resources.

8. Pre-test the questionnaire

Even after the researcher has proceeded along the lines suggested, the draft questionnaire is a product evolved by one or two minds only. Until it has actually been used in interviews and with respondents, it is impossible to say whether it is going to achieve the desired results. For this reason it is necessary to pre-test the questionnaire before it is used in a full-scale survey, to identify any mistakes that need correcting. Usually a small number of respondents are selected for the pre-test. The respondents selected for the pilot survey should be broadly representative of the type of respondent to be interviewed in the main survey.

9. Develop the final survey form

If the questionnaire has been subjected to a thorough pilot test, the final form of the questionnaire will have evolved into its final form. All that remains to be done is the mechanical process of laying out and setting up the questionnaire in its final form. This will involve grouping and sequencing questions into an appropriate order, numbering questions, and inserting interviewer instructions.

Physical appearance of questionnaires

The physical appearance of a questionnaire can have a significant effect upon both the quantity and quality of data obtained. The quantity of data is a function of the response rate. Ill-designed questionnaires can give an impression of complexity, medium and too big a time commitment. Data quality can also be affected by the physical appearance of the questionnaire with unnecessarily confusing layouts making it more difficult for interviewers, or respondents in the case of self-completion questionnaires, to complete this task accurately. Attention to just a few basic details can have a disproportionately advantageous impact on the data obtained through a questionnaire.

Structured Interview Design

One key element in conducting useful research is gathering reliable information. And the basis for doing that is designing questionnaires that get the kind of information from which the researcher can draw valid conclusions. This part looks at some of the basic building blocks of a structured interview, points out potential pitfalls, and suggests ways for the researcher to avoid them, in order to produce a set of questions that have the best possibility of generating reliable, accurate data on the topics of interest.

1. Some basic terms

The first thing that a new researcher needs to know is some standard terminology. This knowledge helps that person understand other concepts to be introduced and puts him or her in a position to communicate with more experienced people in the discipline.

·Structured interview—One that uses a Data Collection Instrument (DCI) to gather data, either by telephone or face to face. In a structured interview, the evaluator asks the same questions of numerous individuals in a precise manner, offering each individual the same set of possible responses. (In contrast, an unstructured interview contains many open-ended questions that are not asked in a precise, structured way.)

·Computer-assisted telephone interview (CATI)—A form of telephone interview where the DCI is stored in a computer and the interviewer records responses directly into the computer.

2. Designing a structured interview

The first step in designing the interview is to formulate the broad overall questions that the survey is intended to answer, i.e.

·Why is the study being undertaken?

·What does the study aim to learn or determine?

The second task is to translate the broad overall questions into measurable elements as hypotheses or more precise questions.

Then, the target population needs to be identified. If computer-assisted math instruction is new only to a certain grade or school, care needs to be taken to interview only the parents whose children have been affected by the program change. It may sound basic, but innumerable studies have proceeded to unsatisfactory completion because of insufficient consideration of this aspect.

Now, the study can proceed to the development of a pool of specific questions designed to elicit the desired information. The number of questions developed should be more than the number to be asked; then, the most appropriate and useful can be selected from those available.

3. Composing appropriate questions

Three main criteria exist for writing appropriate questions: relevance, selection of the proper respondents, ease of answering.

·Relevance—Questions should be directly related to the purpose of the study and have a good probability of yielding the kind of data desired.

·Selection of respondents—Even though a question may be relevant to the study, it may not be answerable by the people to whom it will be asked.

·Ease of response—Questions need to be relatively easy to answer and should not create embarrassment for or an undue burden on the interviewee.

4. Selecting a question format

Important considerations in deciding on the format of questions include how the question is to be delivered (mail, telephone, face to face), the type of information the respondent is expected to provide, and the possible alternative responses. Making these decisions will result in the selection of open-ended, fill-in-the blank, binary-choice, scaledresponse, or unscaled-response questions. Of course, depending on the type of information desired, a structured interview questionnaire will generally have a combination of these types of questions.

·Open-ended—Because open-ended questions provide no structure for the answer, they should be tightly focused to elicit the kind of information the researcher wants to get. And, because they require accurate and time-consuming transcription, their use should be limited to initial research where the number of respondents is small and the object is to refine the research direction and determine more precise questions that can be structured another way.

·Fill-in-the-blank—This type of question has a simple answer, usually a name, frequency, or quantity, which is the kind of information these questions are good at obtaining.

·Binary—These are good for obtaining factual information that falls into the yes-no, true-false category of answer.

·Scaled-response—These consist of a list of alternative responses that increase or decrease in intensity in an ordered fashion. These kinds of questions can be further defined as balanced, unbalanced, and rating and ranking.

·Unscaled-response—With this type of question, the respondent is asked to choose one or more options from a list; this is the type of question that should include the other category so that the responder is not forced to select an answer with which he or she is not completely satisfied.

Activity 2-1: Finding out the similarities and differences between two research design methods

Read the section again and work with your classmates to find the commonalities and differences between the two research design methods: questionnaire and structured interview. Then finish the following chart.

Activity 2-2: Recognizing the purposes of pre-testing the questionnaire

Read the sentences and tick “√ ” after each sentence which is true about the purpose of pre-testing the questionnaire. Make notes what helped you reach your decision.