|
With questionnaire-based studies you usually have some or all of the following issues to deal with and report in the :
| 1 |
processing the data prior to analysis - possibly including the coding of open-ended questions |
| 2 |
reporting the response rates |
| 3 |
problems with missing data |
| 4 |
perhaps having multiple sets of DVs and analyses to report in sequence |
Often you will need to derive your DVs by computing scores from the responses to two or more individual items. For example, you might calculate and analyse the mean score per participant across a set of ratings. Under these circumstances you should really compute and report a statistic to demonstrate that the items that you combined do appear to be reliable –
to “hang together” and measure the same thing. A commonly used statistic for this is Cronbach’s alpha and you can find more about this in Section E9 of this Web site. Report this measure either here or in the (but not in both sections). Calculating Cronbach’s alpha may have led you to exclude some items that you had intended including in your combined measure and you would need to report the details of what was included and what was left out. Technically, these are issues to do with the internal consistency of the measure you are using. That is, the extent to which the items are consistent with each other and appear to be measuring the same thing.
The questionnaire may have included open-ended questions. These are ones in which you do not provide response scales but allow participants the freedom to respond in their own words. Where relevant, you may have got two or more people, blind to the hypotheses of the experiment, to code these responses in some way, giving you numbers that you can analyse inferentially. (For more on inferential statistics, see Section 4.2 of the book.) You need to describe how this was done, the coding scheme you used (and how this was developed) and some index of interrater reliability. That is, a measure of the extent to which two or more raters classified the content of the responses in the same way.
If the data provided by these questions are ancillary to the main issue, then this description can come later on in the . If they are central, however, this process needs to be described in the opening paragraphs to this section.
You may well have handed out more (perhaps many more) questionnaires than were returned. You need to report the response rate, which is the proportion of the questionnaires handed out that were returned satisfactorily completed. Express this as a percentage. Low response rates may pose problems for the external validity of your study. (For more on external validity see Section 10.8 of the book.) In experimental studies involving questionnaires, however, you should also check that there were no significant disparities in the rates of return of questionnaires in the different conditions. This is just one example of mortality effects. This is a grim term for a grim problem. Mortality effects are the effects of your condition on recruitment, completion or retention of participants. In medical trials mortality effects are literally that – the rates of people who die in each of the conditions. Fortunately, participants do not usually die as a result of our experiments in psychology, but some proportion will be unattracted by your description of the condition they are randomised to and decline to take part, or find the task sufficiently uncongenial that they give up during it or otherwise fail to respond subsequently if your study has a later follow up. The main problem for experiments occurs when these rates differ systematically between conditions, as this can affect both the analysis and interpretation of the findings. If there are disparities between conditions this may indicate something about the nature of the conditions and can heavily qualify the conclusions you can draw from your experiment. Differential mortality affects the internal validity of your experiment. (For a discussion of internal validity see Section 10.9 of the book.) When you come to enter the data from your questionnaires, you will invariably find that participants have missed out questions, perhaps whole pages, or given responses that are ambiguous (perhaps circling two responses where only one was expected). These missing data may also turn out to be systematic, affecting some of the questions significantly more than others. If so, you should mention this problem. Missing data are another instance of mortality effects and may also affect how you analyse the data.
Where you faced any of these problems, you should describe what you did about them in the opening paragraphs to the section. (See Section H1 of this Web site for more on what to put in the opening paragraphs of more advanced sections.)
Depending on the nature of the questionnaire, you may have a whole series of DVs that you wish to analyse and even distinct sets of IVs. Some of these IVs may not be independent variables in the sense in which I define these in Chapter 9 of the book. Variables like sex, social class, whether people smoke or not, are for or against abortion, and so on, can often look like IVs but in fact they are not experimental. For a discussion of this issue and its implications, see Section 13.8 of the book and Section F of this Web site. Most of the time this issue will not affect the way in which you analyse the data inferentially. It will, however, substantially influence the conclusions you can draw from your analyses.
As ever, where you have a series of analyses to report, you should deal with them in meaningful units and work through them in a sensible sequence. (For more on this see Section 4.3 of the book).
However, despite appearances, your questionnaire may not lead to such complex analyses. This is because you may be doing either or both of the following:
| 1 |
ignoring much of the questionnaire content in the and only focussing on the key parts - the IVs and DVs. That is, much of the questionnaire may in fact have consisted of filler items. These are items that are there to camouflage the key questions and disguise the intent of the researcher. You should describe such items in the but you can ignore them for the purposes of analysis. |
| 2 |
combining responses to questions into composite measures. For example, you may have worked out the mean or total response per participant to particular types of item, or calculated scale scores or scores on particular factors of the questionnaire. This can greatly reduce the amount of data that you have to report in the . (For more about combining items in this way see Section E9 of this Web site.)
|
If you have used scales that have been developed and published elsewhere, these will often have sets of norms. These are the known scores on the scale for certain populations, such as a particular age group or educational level. It is useful to relate the scores of your participants to the most relevant set of published norms to determine whether your sample has performed in the expected way or whether it deviates from these norms. Again, you can do this in the opening paragraphs to the . If the sample does deviate, then you will need to examine the possible reasons for this in the . It raises the possibility that there is something anomalous about your sample or about the use of the scale in the context in which you employed it (e.g., perhaps it was developed in another country and does not function in the same way in your own).
Sometimes in more advanced studies you may wish to examine the effect of say a manipulation some time after the initial exposure. Often such measures take the form of a brief self-report questionnaire given to participants at a later date. Unless it is very different from the initial questionnaire, you can describe this follow-up measure and its administration at the end of the and respectively. |
