Generalizations 10# Kobayashi, S. (2018)

At the current stage this project is about finding statistical generalizations in qualitative, danish educational research. That is to check my assumption that there is a tendency to make statistical generalizations on the basis of observations which cannot support such generalizations.

I will be posting excerpts from all those studies which to me seem problematic. This post is part of that and this is a link to the first post in the line.

Kobayashi, S. (2018)

Reference:

Kobayashi, S. (2018). Peer feedback among international PhD students. Dansk Universitetspædagogisk Tidsskrift, 13(25), 91–106.

https://doi.org/10.7146/dut.v13i25.104668

Abstract:

In a PhD course for new PhD students peer feedback was introduced to reduce teacher time on feedback and to enhance the learning environment. The results of the changes to the course are not conclusive with regards to teacher time, since there were also oth-er changes made to the programme, but overall teacher time on giving feedback has been reduced. Peer feedback in higher education is seen as one way to enhance the learning environment for students as it builds on principles of formative feedback dur-ing the course of study and when students give feedback it has been shown to enhance learning. The results from this study support this view, but improved learning was only observed after peer feedback was integrated in teaching and learning activities em-bedded in the course rather than as an add-on.

This article describes and evaluates the introduction of an element of peer feedback in a PhD course. Peer feedback was introduced with the double goal of saving teacher time and enhancing learning outcomes. The changes made to the course were initiated as a development and learning project undertaken as part of my participation in the Teaching and Learning in Higher Education Programme (Universitetspædagogikum) in 2016. The aim of the article is to share experiences that indicate that this double goal is achievable when a) assessment (or feedback) criteria are explicit and shared and b) peer feedback is an integral part of the course.

 

Testable hypothesis?:

Ja, i princippet: "Peer feedback was introduced with the double goal of sav-ing  teacher  time  and  enhancing  learning  outcomes." -abstract. 

Så som hypotese noget i stil med "peer feedback sparer lærertid og øger læringsresultater.

Dog: "The aim of the article is to share experiences that indicate that this double goal is achievable when a) assessment (or feedback) criteria  are  explicit  and  shared  and  b)  peer  feedback  is  an  integral  part  of  the  course." -abstract

Vores ritual virker kun når disse specifikke/abstrakte ting er på plads på den helt rigtige måde.

 

Method/materials:

"Comparing the PDP assignments submitted with earlier assignments, to see if it is possible to judge whether the quality increases. 6.Constructing and distributing a questionnaire to get feedback from partici-pants after the assignments have been approved, to learn how they per-ceived the peer feedback." s96

"our (out of 23) were asked to resubmit, at least three were inade-quate, but were still acceptable, and around five were really good with substantive thinking reflected in the writing." s98

" I distributed a questionnaire to get the students’ experience of how the peer assessment worked. I received eighteen responses to the questionnaire from the 23 participants. " s99

"Of the 18 respondents, 60% found it meaningful to give peer feedback while 17% found it difficult, and another 17 % did not give peer feedback (two found the techni-calities of the LMS to be a barrier and two were not confident that they could pro-vide good feedback). Similarly, 60% found the feedback criteria helpful, while 27% found it difficult to use the criteria. Reviewing other PDPs seemed to help the vast majority." s99

 

Statistical generalizations:

1) "Overall, this first iteration of using peer feedback in the Introduction course did not seem  very  successful  in  terms  of  higher  quality  assignments  and  less  need  for  teacher feedback. Still, the analysis of the sample of assignments and peer feedback, and  a  questionnaire  distributed  to  participants,  indicate  that  peer  feedback  has  the potential to  support  learning  in  the  course. " s99

2) "The improvement of assignments after integrating peer feed-back  reflects  an  enhanced  learning  environment  where  course  participants  learn  from  each  other.  The  reduction  in  the  number  of  assignments  requiring improve-ment means that teachers spend less time on feedback- a personal estimate is that the  time  spent  on  feedback  can  be  reduced  from 40-60  minutes  per  assignment  to  20-40 minutes per assignment. " s103

 

3) "The tentative conclusion is that it is possible to both increase financial sustainability and  enhance  the  learning  environment  through  the  use  of  peer  feedback  on  as-signments, if formative feedback is taken a step further to formative assessment by making  feedback  criteria  clear  and  shared,  and  making  peer  feedback  an  integral  part of the course so that the feedback loop is complete and course participants are active partners in the feedback process." s103

Comments:

I have flagged this study for generalizations of type 4 and 5 (see typology).

 

The last sentence of the first quote represents a very modest and careful generalization. Careful to a degree that we might even agree that it is not really a generalization. As i've noted before regarding similar cases, it is nonetheless important to discuss whether great care is enough? Is it proper to merely suggest a possible generalization if the data cannot be made to support it?

The second quote is an example of a direct generalization. The study is said to support two claims ("...enhanced learning environment..." and "...means that teachers spend less time...") that it cannot support, neither by method, nor by number of observations, the latter being the focus here. 

The third quote is a generalization of the present study, implied in an ethical-to-do-list. The items mentioned (eg. clear, shared criteria, participants as active partners) have the effects mentioned ("...increase financial sustainability and enhance the learning environment..."), insofar as the results of the study are generalizable to similar situations and populations in the future. It is not however.