At the current stage this project is about finding statistical generalizations in qualitative, danish educational research. That is to check my assumption that there is a tendency to make statistical generalizations on the basis of observations which cannot support such generalizations.
I will be posting excerpts from all those studies which to me seem problematic. I will do this for two reasons: Firstly, there is no consistent method of identifying unwarranted, statistical generalizations, which is not ultimately dependent on my judgement. For this reason it should be possible to review my judgement. Secondly, many of the generalizations that i have identified, do not feel problematic to read for the simple reason that many researchers do use careful language, which is a good thing. It is important to be able to recognize that while a generalization may be technically problematic, it can be delivered in such a way, that most people would not find it problematic. Therefore it must also be possible to discuss whether such cases are or are not problematic. Does proper delivery excuse the problem? There are certainly many cases of research coming under criticism, not because it is bad research, but because it was sold (in media) as being more than it was.
Anyway this is the first of the studies that i claim to be problematic:
The study
Reference:
Langhorn, R., Bjørnvad, C. R., Fog-Larsen, A. M., Willesen, J. L., May, M., & Langebæk, R. (2018). A Virtual Veterinary Emergency Clinic – investigation of students’ perceptions and self-efficacy beliefs. Dansk Universitetspædagogisk Tidsskrift, 13(25), 120–133.
Abstract:
"At the University of Copenhagen, companion animal emergency medicine is taught in a clinical environment after students’ completion of basic theoretical and clinical courses. Students are often anxious about emergency shift partici-pation and the prospect of being the responsible veterinarian in emergency sit-uations. This study aimed to investigate whether inclusion of virtual patients in addition to real-life patients would increase students’ perceived self-efficacy in emergency medicine. Sixty-seven students were divided into two groups, one of which participated in regular emergency rotations, while the other also learned in a Virtual Emergency Clinic (VEC). Participating students were given a ques-tionnaire regarding course experience and self-efficacy, with responses on a 10-point Likert scale. The VEC group expressed a higher level of knowledge and sig-nificantly higher level of exposure to and ability to handle emergency patients. In addition, virtual problem-based learning appeared to increase veterinary students’ self-efficacy with regard to managing emergency patients in their fu-ture careers."
Testable hypothesis?:
Yes: "The aim of this study was to investigate whether inclusion of virtual patients in addi-tion to the real-life emergency patients at the UHCA would lead to an increase in students’ perceived self-efficacy in dealing with emergency patients. "
Method/materials:
"Twenty-nine students participated in the first EOCA course in the spring of 2015 and were designated to the 'No VEC' group. This group completed the course in the tradi-tional manner based on patient intake in the hospital and with no access to the VEC. Thirty-eight students participated in the second EOCA course and were designated to the 'VEC group'. During the introduction week, a thorough demonstration of the e-learning program was given, and written instructions were supplied. Each student was to access the VEC on-line and complete four virtual patient cases, write a medi-cal record for each case and send these to the course instructor before the end of the rotation. Prior to the EOCA course both groups had participated in a general practice course at the same hospital, but had not yet encountered critically ill or emergency cases. All students volunteered to participate in the study, which was approved by the Veterinary Study Board at the University of Copenhagen. Three stu-dents who participated in the course prior to initiation of the study had volunteered to be interviewed before and after their course in order to clarify students' percep-tions of their self-efficacy in relation to emergency medicine and to obtain infor-mation that could identify key subjects to be included in a questionnaire." s123
Statistical generalizations:
1) "The results of this study demonstrate that including problem-based learning in the form of a virtual clinic in the EOCA course increases the self-efficacy of Danish veter-inary students for managing emergency patients under supervision during their training and indicate a similar effect with regards to patients in their future careers. We believe that this is likely to be true for veterinary students internationally. " s130
2) "This encourages us to priori-tize this educational tool, add more cases and further improve the VEC so that, in the future, it will hopefully simulate the emergency room so closely that only the live animal is lacking." s131
Comments:
I have flagged this study for generalizations of type 4 and 5 (see typology).
The first example is a direct generalization, mostly due to the last sentence. The language is a little careful, since the researchers describe the generalization as a matter of their belief ("We believe..."). The study does not however hold enough observations for them to be generalized beyond the borders of Denmark. Perhaps they cannot even be generalized beyond Copenhagen. The first sentence in the first example can also be seen as problematic, mostly due to the fact that the study is self-report. For this reason it can only indicate a change in opinion of self-efficacy. One might argue that self-efficacy is a validated construct, but this would only apply to my critcism had the authors of the present study utilized one of the validated inventories. The questionaire of this study however comprises four items, only two of which are actually sentences that could be used to indicate self-efficacy. As such the sentence: "The results of this study demonstrate" is simply too strong and borders on misinformation.
The second example is less problematic. I note it only to indicate another shade of the same problem. Whether the studies i have read for this projects, include type 4 generalizations or not, the researchers nonetheless often make comments which are not in themselves statistical generalizations, but that imply a belief that the results of the study are in fact statistically generalizable. The second example, as noted here, is not a very grim case of this situation. Yet it implies that the results of the study are strong enough, that they warrant further development of the educational tool that was tested. It only makes sense to make such a statement, if it is implied that the tool in question will have similar effects on similar populations in the future. In other words the statement rests on a statistical generalization.
Ingen kommentarer:
Send en kommentar