Working with generalizations

 The following is an excerpt from an article i am working on, as such it is written in english. One of the hypotheses that i am interested in concerns the use, in qualitative danish educational research, of unwarrented generalizations. That is generalizations made as part of the conclusion of a research article, which are not supported by the knowledge produced in the article. The point is not to say that this is a problem in itself. That is to say, it sounds bad, but i have learned, through reading many examples, that wording does mean a lot. 

Anyway i have developed an adhoc typology of generalizations which will be relevant to reading many of the things i post here. It is messy because i actively use it when registering in my dataset, and for this reason any changes are very time-consuming.

The text:

Types of statistical generalizations in danish qualitative, educational research articles, presented in two journals (DUT and SLP):

Type 1: General statements made on the basis of the results of the study, that are not unwarranted.

Type 2: Unwarranted general statements not made on the basis of the results of the study, but with reference to one or more other studies.

Type 3: Unwarranted general statements made on the basis of the results of the study, but interwoven with a claim about one or more other studies that seem to support the same findings.

Type 4: Unwarranted general statements made on the basis of the results of the study.

Type 5: Unwarranted general statements made on the basis of the results of the study, presented in the guise of ethical guidelines or suggestions for practice.

Type 1 is, as stated in the definition, an anti-type, used in those situations where no unwarranted general statement was made on the back of the results of the study. Suffice it to say, this type was not observed.

Type 2

Type 2 generalizations while probably the most observed type, was not used often. Checking every general statement made with reference to other studies, was not central to satisfying the hypotheses here dealt with. (EXAMPLE - Literature study)

Type 3

Type 3 can also appear as an example of type 4, that is “a direct generalization”. What determines the difference is whether the generalizing statement refers to the results of other research, and not just the research of the article itself. One might argue that this must certainly be unproblematic, and that on the contrary, it must be the duty of the researcher to point out similarities between their own research and other research. And indeed, this is why we have literature reviews. It is always relevant for researchers and readers to know what other research exists on the subject of an article they are writing or reading. However, the results of a study do not become more generalizable, just because a similar study exists. The plausibility of such a statement and the generalizability of the study in question, can only be claimed with a proper description of the similarities of the studies involved. Especially when we are talking about qualitative research. We cannot simultaneously hold that people answering a questionnaire, to determine their personality type, may mean completely different things with the same answer, while also maintaining that case-studies have similar results, if only the results look alike. Two case-studies made about the same subject, may not be comparable at all. In coding they may vary greatly simply because the studies were made in different countries. To make a warranted type 3 generalization in a study that is not generalizable in itself, at least two criteria must be met:

• It must be clear precisely why and how the other studies involved are comparable.
• The other studies involved must contain at least one study large enough to be generalizable, or the aggregate of the studies must be generalizable. 

Note that a study need not be giant if the population that any generalization is concerned with, is not giant. If there are only a hundred researchers of education in Denmark, we should not need to interview all of them in order to make generalizations about the rest. However, such generalizations would still depend on the types of information we are trying to collect. 

Example of a type 3 generalization:

Oksbjerg, M. (2022) poses the following research question:

“Which conceptions do teachers of the subject danish, hold regarding the formation [(from danish: “dannelse”, also often expressed with the german word: “bildung”)] of students when it comes to teaching about literature, and how do the teachers see these conceptions supported in didactical learning materials, that they choose to teach with.” -Oksbjerg, M. (2022:77)

The article is based on four interviews with teachers of the primary school subject danish. They all used different parts of the same digital, didactical learning material, called Fandango (digital platform) for teaching (Oksbjerg, M. (2022:83)). 

Among other things Oksbjerg concludes:

“Since recent research by Bremholm et al. (2017) shows that learning materials [(digital)] play a central role in teaching the subject danish in primary school, the results of this article can point to a need for supplementary education of teachers in this field.” -Oksbjerg, M. (2022:94)

Through discussion, danish, qualitative researchers might reach the conclusion that a sentence, such as the one quoted, is quite unproblematic. Nonetheless such a discussion must be had, because the sentence in question is an example of an unwarranted generalization. Oksbjerg assumes that the potential, educational needs of four teachers of the subject danish, indicates a need among “... teachers in this field.” as such. This point is made to be supported with a reference to other research (Bremholm), which does not support it. The book by Bremholm et al. is about learning materials used by teachers of danish, but not about their educational needs. Oksbjerg correctly does not cite it as such either, but remarks merely that learning materials “...play a central role…”, which Bremholm et al. does show. Nonetheless the wording of the sentence is: “Since recent research…”, “... the results of this article can…”. In other words the sentence suggests an argumentative interaction between the research referred to and the results of the study in question. Oksbjerg argues further for this interaction on the same page:

“This investigation, as noted, is based on a qualitative case-study, where the mentioned teacher-interviews are included. Within the frame of the conditions of the chosen research design I reason that the results can function as grounds for discussion and further development of both the learning materials of the subject of danish and educational practices. I base this on the information oriented selection of teachers, where differences with regards to educational experience and professional preconditions have been criteria. Moreover the great amount of use, that the specific learning material [(Fandango)] has in danish schools teaching of literature, also adds legitimacy to the results.” -Oksbjerg, M. (2022:94)

The last sentence more overtly suggests the interaction in question. Bremholm et al., as noted before, proves “...the great amount of use…”. Oksbjerg directly concludes that this must add “...legitimacy to the results.”. Even though those results are made to be about, as noted, supplementary education for teachers. Something not covered by Bremholm et al.

Type 4

A direct generalization, in this case, is simply a sentence that either explicitly or implicitly suggests that the results of the study in question also hold for populations beyond the one in the study itself. A sentence that assumes or requires the generalizability of the results of the research in question. As such it is the same as a type 3 generalization, with the exception that type 4 does not include reference to other work in order to sustain the idea of generalizability.

An example of a type 4 generalization:

Drejer, S. N., & Bang-Larsen, A. (2021) looks into what experiences from educational guidance counseling young people bring with them when beginning a bachelor’s degree and what guidance-needs can be identified during the initial year of university (Drejer, S. N., & Bang-Larsen, A. (2021:36)). The article is based on a reanalysis of two case-studies with one student in each, Luna and Veronika (Drejer, S. N., & Bang-Larsen, A. (2021:40)). 

The researchers conclude among other things:

“If Luna and Veronika experience cultural introduction [(from: “indkulturering”)] as a challenging non-linear process, then other students, who to a lesser extent have acquired the content of the subject, will probably also experience cultural introduction as a challenge. Thus we argue that the findings from the analyses of Luna and Veronika are relevant in broader contexts [my translation].”

In this example the authors argue that the analyses of two cases, with two persons, can be statistically generalized. Again, we find two problems beyond the generalization itself. Firstly, the fact that a statement is generally acceptable, does not mean that research supporting that statement, becomes more statistically feasible. That is, unless the research in question is comparable to generalizable research supporting the same statement. Secondly, as with the type 3 example, the problem becomes one of vagueness. Generally, how many people would argue that becoming a part of a new culture, either educational, or work related, or at a societal level, is not challenging? The researchers, whether knowingly or not, make their research seem more broadly applicable, by using it to draw conclusions that most people would accept anyway.

Type 5

Unwarranted general statements made on the basis of the results of the study, presented in the guise of ethical guidelines or suggestions for practice.

To draw conclusions about educational or pedagogical practice, on the basis of the results of a study, implies that those results are expected to reappear in future situations. Therefore such conclusions are implicit examples of statistical generalizations. 

Example of a type 5 generalization:

Beierholm, M. H., Hjorth, M., & Basballe, D. A. (2022) pose the following research question, regarding a new subject being introduced into primary school, and primary school teacher-education, called “technology understanding” (my translation):

“How are the areas of competence of the subject Technology Understanding to be integrated into classical teacher professionalism, so that Technology Understanding becomes partly a fundamental part of teacher professionalism and partly a contributor to the “formation” [(“dannelse” or “bildung”)] of those who study to become teachers. -(Beierholm, M. H., Hjorth, M., & Basballe, D. A. (2022:13))

The study is based on a course run with a class of teacher-students. The materials include observational notes and video. The results however are primarily derived from a report written by the students as part of the course. The researchers received 22 reports from the students, which is the best indication in the article of the actual number of students. From those 22 reports only 8 were used as part of answering the research question of the study, since those 8 included problems related to Technology Understanding-professionalism (Beierholm, M. H., Hjorth, M., & Basballe, D. A. (2022:21)). 

The researchers make the following type 5 generalizations:

“Finally [conclusion, not an exclamation] it looks as if it could be potentially fruitful with teaching as an interaction between use/experience, which leads the students astray, and production/experience, which leads the students behind the technology, as a way toward digital empowerment and formation” -(Beierholm, M. H., Hjorth, M., & Basballe, D. A. (2022:27)). 

The researchers make a very careful generalization, by suggesting a certain way of teaching when it comes to digital empowerment. It becomes a statistical statement because of the suggestion that, due to the results of the study, it is probable that students who, in the future, experience the interaction described, will become more digitally empowered. 

This is, as noted, an example of a very careful generalization, and it is of course appropriate to discuss and question whether it is unproblematic to make unwarranted generalizations, if only the language used is sufficiently careful? Perhaps the argument that researchers should communicate their work very carefully can, if enforced too rigorously, become an obstruction to language as such.