Published on 12, July, 2020
On a late-night (early-morning?) internet trawling sessions the other day, I came across a blog I hadn't seen before (link at the foot of the post), which those of you interested in the ethics and other standards behind research into autism may find interesting.
It is written by an autistic Canadian woman, who despite not being formally qualified as a researcher, has been getting up to her elbows in the topic, and in particular the way that research into autism seems to be held to less stringent academic standards than other areas of psychology research - notably when new interventions are introduced which make wild claims, often before they have been properly validated, so tainting the research into them.
I have yet to read all of it myself, and am unsure that I agree with everything, but it raises some very worrying points about the quality of autism research. Unfortunately, most of the referenced papers are behind paywalls, which is rather a shame. Also note that when she talks about "autism advocates", she usually doesn't mean grass roots advocates, but organisations such as Autism Speaks; which rather confused me at first.
The Autism Crisis (Michelle Dawson)
I highly doubt it has lower standards if you consider the research in top academic journals, such as Science, Nature, PNAS, Neuron, Journal of Neuroscience, Cerebral Cortex, Current Biology, TICS, eLife, etc. However, I do agree that there are some Autism-specific journals that have lower IF scores and lower standards in getting a paper accepted, but many other theme-specific journals (e.g., aging, addiction, ADHD, etc) have the same issue, as well as other generic journals with low IF.
I think it's in the specific area of interventions that she's most concerned, as opposed to theoretical research. What she seems to be saying is that interventions are being introduced in practice before peer-reviewed research has even been completed, and that because implementation and "marketing" to service providers is coming before, rather than after, independent validation, this impairs the ability to perform truly blind randomised clinical trials.
As I say, I'm still digesting it myself, so I'm not wholly convinced, which is partly why I thought I'd throw it out there to get some alternative opinions. I certainly haven't discounted yet that she may have a personal axe to grind of some sort.
I think we're on the same page then? It's not the trials of interventions that bothers you? As I said, I don't see how you can decide whether an intervention is useful or not without trying it. It's rather the making sure that the methodology used in the larger, more controlled trials, gives scientifically useful results?
By conducting randomised clinical trials - the key word being "randomised". True double-blinding is, of course, much more difficult with these kinds of interventions than, say, a pharmaceutical, but certainly would not be helped by promoters of the intervention having already widely spread their unproven claims and methodologies such that those performing the trial are influenced by confirmation bias because they can identify the intervention and have come into contact with unverified claims.
The blogger talks about this particularly towards the start of this post and those linked from it.
Having said that, if interventions and publicity are made widely available to service users (other than those taking part in trials) before they are verified, this isn't a problem of science research as such, it is one for administration of health-care systems, those who determine what research to fund, and the media, and I do think that this is a weakness in some of the arguments in the wider blog - she seems a little over-keen to blame researchers and the constraints that they work within rather than those who are misreading what their research is or is not able to tell us.
How do you "research" interventions without trying them? I would think the "research" would be comparing potentially favourable interventions in larger controlled trials based on ones that looked promising in smaller trials less controlled trials.
I haven't yet read all of her blog entries as it's quite long, but started from the top few. I thought you were referring to that one in particular since it seemed most direct in saying that she thinks research into autism seems to be held to less stringent academic standards than other areas of psychology research.I've now looked at a few more posts about her thoughts on specific papers, and I think a few share similar problems to the previous post that I criticised. And I agree with her on some of the posts on deficiencies of some of the papers and research questions. And I think it's good that she has noted these and compared results from various studies showing contradictory or vague conclusions. But I think that's just how science is at the moment, in any psychology research domain, it's not limited to autism research. There are just a lot of things we don't know yet, and since the data are collected from people and different designs, there will be variability across studies. That's natural. So we need more and more research over years to clarify why some studies show different results, and refine current hypothesis that are vague or incomplete.
Sorry, I think I really should go to bed, I got a bit flustered there when we were both, I guess, typing at the same time! Oh, and I should also add, my 'A' levels were maths, physics, chemistry, so I'm enjoying getting stuck into some new fields since my diagnosis, and I'm glad to have more keen minds such as yours to offer some guidance. I think I shall read your previous comments in more detail after giving my brain a little rest. Thankyou again for you efforts.
I've now had a look through the 'influential paper' she cited: https://link.springer.com/article/10.1007%2Fs10803-007-0517-7The standards don't seem specifically low at all. All of the standards they listed were good rigorous scientific practice, and seems similar to what one would expect when conducting intervention research on any population. I would say some of the criteria sometimes can be even hard to fulfill completely in non-autism research (e.g., controlling attrition rate; evaluating generalization and/or maintenance by collecting measures after final data collection). In fact, the paper aimed to aid in narrowing the gap between research knowledge and applicability of research in real world settings. It's published in an autism journal, but the guidelines for good science seems quite general. You can compare the definition of group research quality indicators in Table 1 of their papers with Gravetter, F. J., & Forzano, L. B. (2012). Research Methods for the Behavioral Sciences (4th ed.). Belmont, CA Wadsworth. which is a textbook for general psychology research methods (not specific to any population), and the key indicators are pretty much the same.
And, actually, the original sentence about generality of evaluation method in the paper was: "Although these instruments were developed and field tested on interventions for young children with autism, there is no reason the definitions cannot be modified to allow their use by professionals who work with other populations (e.g., ....)". Logically "other populations + e.g.," means that it is inclusive of the typical population. It also suggested using this standard for populations like mental retardation and Down syndrome, so I don't know what she's complaining about.
I agree it is important to evaluate the evaluation standards for autism intervention. And it's good to keep improving research quality and applicability to the real world. And it would be a huge problem if autism intervention research is of much poorer quality than other intervention research. But this needs to be evaluated scientifically. It would be more convincing if she had been able to compare the evaluative method suggested in this paper with those of general and non-autism intervention studies. And from what I have seen so far, I don't think the evaluation standards are much different.
Ah sorry, yes, I haven't made something clear here. That post is her, say, overarching conclusion, but the whole of the blog is essentially based around the same premise (and of course, is in reverse chronological order.) I have to admit that I found that particular post rather hard to parse, as opposed to some of the other more concrete examples in other entries. My apologies for not being clear enough.
All the same, your analysis is appreciated - these kinds of writings do take quite some time to sink in for me, as my formal science training didn't go much beyond 'A' levels. I would certainly think of myself only as a fascinated amateur!
And, just to add, I think there are just a lot I don't agree with her in a lot of her arguments.
She claims that autism intervention research have lower standards than non-autism areas, she hasn't cited or listed the interventions done with other clinical populations. Furthermore, she hasn't cited a single review/meta-analysis paper that has suggested lower standard autism research. So other than strong wording, there's no concrete convincing evidence. Furthermore, I have seen a lot of other low standard research from non-autism areas.
I also don't agree with her logic. She says "In one influential paper, the authors devised an elaborate system of autism-specific low standards for evaluating intervention research, then suggested extending this segregated system to other developmentally disabled individuals." So, based on this statement, it means that non-autism research on clinical populations (at least 'other developmentally disabilities') should have similarly ''low" standards. That is directly contradictory to her claim that autism standards are lower than non-autism standards. Logic doesn't work out.
And the reason that the authors do not suggest doing this to the typical population could easy be that there are no interventions done for typical people who are not struggling with their lives.
Trogluddite said:What she seems to be saying is that interventions are being introduced in practice before peer-reviewed research has even been completed, and that because implementation and "marketing" to service providers is coming before, rather than after, independent validation
I don't see where she says that...
This is the post you are referring to, right?
Autistics have long been subject to separate, unequal standards in both research and practice. Claims that standards must be lower for autistics than would be acceptable for anyone else continue to prevail. When it comes to autism interventions, autism-specific low standards have proliferated, by popular demand, and are now nearly universal. History tells us to recognize this as segregation at its insidious worst, but this has not yet happened.
Instead autism organizations and advocacy leaders, including those known for opposing each other, agree when it comes to segregating autistics this way. In their documents, policies, recommendations, and lobbying, they all promote autism-specific low standards—especially in interventions. Ignoring the strong lessons of history, they all support the segregation of autistics from the basic rights and standards which protect and benefit everyone else. Their high-profile disputes are, instead, over which kinds of bad science and bad ethics should be foisted on autistics.
The results are obvious. In non-autism areas, poor quality research and its harms—its waste of resources, its misleading findings—are vigorously condemned. There is recognition that even the best existing research standards are flawed and need always to be improved. But when it comes to autism, standards have instead been lowered or discarded to accommodate the extremely poor autism intervention literature. Poor standards in intervention research are seen not as harmful and wasteful, which they are, but as what autistics need and deserve. Resources have poured not into improving these abysmal standards, but into making the very poor quality autism intervention literature more powerful and influential.
In one influential paper, the authors devised an elaborate system of autism-specific low standards for evaluating intervention research, then suggested extending this segregated system to other developmentally disabled individuals. The authors did not suggest doing this to the typical population, to people like themselves—this clearly would be harmful and unacceptable.
The segregation of autistics from basic standards in research has lowered the bar for standards applied to autistics in other areas: in practice and professional ethics, in public policy, politics, law, journalism, medicine, education, and the list goes on. It has further removed any incentive to fund, conduct, or publish good quality autism intervention research. Yet, the popular effort to impose autism-specific low standards in the area of intervention research can be shown to lack any scientific or ethical foundation.