Critiquing Qualitative Research Articles
by Mark Firth
Research problem, question or hypothesis
Is the problem or hypothesis clear and concise? It appears that no specific research questions are actually stated. Again this poses a problem because we know what the researcher is going to do but the specific questions that need answering are not explicitly written. The purpose of this ethnographic research is not necessarily to generate hypotheses but rather to assist practitioners to conceptualize learner difficulties within a model framework. Similar to the characteristics of a case study the approach has a specific idea of what it wants to find out and from whom. The article could have expressed its questions of investigation something along the lines of:
- From their own perspectives, what real-time listening comprehension difficulties do English as a Foreign Language (EFL) learners face? And.
- At what stages of the language comprehension model according to Anderson (1995) do these difficulties occur?
What type of research is being carried out? Flecks of the research description resemble the processes often associated with quantitative research. The deliberate use of constructs to acquire and analyze data for immediate interpretation for example tells us the researcher not only knows what information she needs but almost hypothesizes as to what the outcomes will look like. Qualitative research does not usually impose structures on the data until the implications of the data begin to ‘emerge’ Punch (ibid:p.25). And so, such a vivid conceptual framework and design usually tends to be associated with quantitative research.
Without going into too much detail about methods, there is however a concern to reveal the subjective beliefs of the learners being studied and an investigation into ‘definitions of the situation’. These purposes are attributed to the approaches of participant observation and ethnography within the realm of qualitative research as described by Wainwright (1997) in the defining of qualitative research.
A lot of flexibility is given to the qualitative researcher with regards to methodological issues as Punch (ibid:p.161) tells us with regards to ethnographic research. Continuums of data collection techniques exist from non-participant observation to ‘words of the natives’ participant descriptions. In fact it seems as though almost any kind of technique can be used in an attempt to gain a ‘fuller picture of the live data’.
Again to categorically label the type of research would limit the potential value of the research and we need to take it for what it is worth. It could be said that the researcher in this instance has culminated the benefits of a ‘tight’ design, employing ethnographical data collection methods from a group of case study-like participants. From the outset it is evident that the researcher has generalizability in mind as the study is intended to inform teachers of how they can better assist learners by knowing where listening difficulties can occur Goh (ibid:p.57).
Is the population described adequately? As a population, ESL students are identified as learners who at different stages of listening proficiency experience various difficulties.
With regards to the confines and purposes of this study it could be argued that further description of the population is not required. However, one point that could be worth mentioning is that we don’t really know of the differences (if any) that exist between learners of different first languages.
Is the sample clearly described? Whilst the article doesn’t go to great length to describe the sample, for all intensive purposes of this study the information could be said to be adequate. That is to say with regards to the sample there doesn’t appear to be a large number of extraneous independent threats, which could affect the results.
Is the method of selection of the sample clear? It is not mentioned as to how or why 17 of the 40 students were chosen or volunteered to participate in the small group interviews, nor how 23 students came to take part in the ‘retrospective verbalization procedure’. While logically it would appear that in this kind of study this should not affect the results, it would have been more comforting for the reader to have such details.
Could the method of selection affect the results? To answer this question it is first necessary to assertion how indicative the difficulties faced by the sample are of foreign language learners in general. By using the most practical and readily available sample to the researcher, an all-Chinese student sample poses concerns about an important issue of how representative the sample is. It would have been beneficial for the researcher to discuss the peculiar difficulties Chinese learners of English have in regards to phonetics. For instance, the English language consists of certain phonemes not found in Chinese which are difficult to learn, while others are similar to the phonemes found in Chinese but pronounced differently. In general, Chinese speakers of English find English hard to pronounce, and have trouble learning to understand the spoken language Swan & Smith (1987:pp.224-5). Likewise other EFL learners face common difficulties specific to interference from their first language or large phonetic differences Nunan (1991).
The type of data that is to be collected somewhat avoids these difficulties in the sense that it is to be analyzed consistent with the cognitive model generally accepted to apply to the process of developing listening comprehension. In doing so the specific difficulties concerning phonemes peculiar to any given first language can be considered separately in the discipline of phonetics. It could be concluded therefore that it is not likely that the participants’ backgrounds and characteristics will influence the results.
Are subjects likely to be motivated to give biased responses? This would be unlikely. Even if the learners were well aware of what the researcher was looking for this could be seen to only enhance the reliability of the data through the ethos between learners and interviewer. It would however have been enlightening to have some reference to the relationship between the researcher and the participants i.e. we don’t know how frank the students were all the time in the interview sessions.
To page 4 of 6
To the article index