Wednesday, January 6, 2010
Brad Pilon on the Scientific Abstract. Here's my abstract: he's wrong - kinda
Follow @mcphoo
Tweet
I have a lot of respect for the author of Eat Stop Eat. Brad Pilon has but together a lot of well reasoned arguments to for the possible benefits of intermittent ceases of eating (he wishes he hadn't called it fasting) for well being. Great. Today on facebook from a post by Richard Chignel i saw a link about Brad Pillon saying what a scientific abstract is. His thesis is that it's like a Movie Trailer - it's there to get you to buy the Paper. Speaking as a scientist who publishes scholarly papers, this is not my experience nor i suspect my colleagues. So i'd like to take this post to give you a perspective from the other side of the house, going through Pillon's points.
Abstracts Exaggerate? Beyond asserting that the abstract is like the trailer to get you to buy a paper, Pillon also makes some general claims about what's in an abstract. So first problem: a key error is to say an abstract "exagerates" findings in the paper. To generalize like that, a person would need to have a large number of examples. I'm stuck trying to think of one. And i'm a Reader - that's my job title. I read a lot of papers. I read even more abstracts. Exageration in abstracts is not a sustainable practice.
Journals - where scientific papers are published - are managed by editors. Editors of journals, by the way, are almost always also experienced and active researchers. So these editors are also writers. It's their job to make sure what's in the abstract is a fair reflection of the content/findings of the paper. If that were not the case, the journal would soon get a rep for being shoddy. You may say you don't know as a non-expert what journal is shunned or not. There are a bunch of metrics for that, but if you search for "journal rankings" in google, you'll see that there are publically available metrics. And a researcher from any space will make that kind of tour through journals a first port of call, to find out the standing of the publication.
Abstract Structure. This is what i teach our graduate students to consider when they're writing up their results. And if the seminar hadn't been snowed out today, it's what we would have been looking at formally. Let no good lecture go to waste - so let me share this here. The abstract needs at least the following ingredients:
What's the point of this structure? It's actually the opposite in practice of what Pillon asserts. Abstracts are not written to get somone to buy/read a paper, but to help us skip what we don't need.
It's about NOT reading the paper. The point of the abstract is to give one sufficient information to know if they need to BOTHER reading the paper for their area of work. THis may seem kinda odd. IF i've put time into writing this wonderful paper, don't i want everyone to read it? It's not the first thing that hits most researchers' minds.
The hope is that the work is sufficiently interesting or relevant that others will find it so (or find it at all, ever), and thus it will stand on its own merits when folks are looking for related work to their own in that area. These are specialized pieces of writing that assume a certain knowledge level for reading. They are written to advance understanding in a particular field. And their are tons produced annually. Hence people who look at research papers generally do so because they have a question they want to understand.
And when such a question comes up, they usually his a dedicated searchable index site like pubmed or other domain specific indexes - not google or not a journal publisher - to see lists of papers, and to click through to the abstracts, and from there, if desired, one of the links to the full text.
example of pubmed search results
So of this myriad of possiblely relevant information, the abstract acts as a filter.
Of the sometimes hundreds of papers available, which ones should be set aside to read further? That's a first pass. This may sound like that's a match for Pillon's try to get you to read the paper. But authors of this kind of work AREN't trying to get you to read a paper. They aren't trying to show you the best shots from the film. They aren't being highly selective about what they show you. There are those pretty standard, one might say objective conventions, about what has to be in an abstract. No jump cuts. The really dry straight stuff only. We're talking about a community here of people who get to know each other. If you as a scientist started trying to sex up your abstracts, you'd be meat.
And since this is a community practice, we all know that we want to make our abstracts as useful as possible so that another readers can make these determinations. Our own reputations are at stake here, too.
As an aside: another way researchers make determinations about what's important to read in an area: looking at the papers other credible researchers are looking at in that area.
Surveying the Field. The second pass is that the abstracts give a researcher a very good sense of what the trends have been in a particular topic over time. So the researcher can begin to get a sense of what questions have already been considered - very important if that person is planning their own work and doesn't want to reinvent the wheel.
Likewise it's cool to see when a particular topic trended. You can see when a particular question seemed to have been hot by how many papers related to it came out in a year. What happened that it trailed off? Is it now called something else? Is the problem solved?
It's not to get you to buy the paper. Of course there's a difference too between reading and buying a paper. Pillon quotes the price a publisher assigns to a PDF. That's the publisher. If anyone is thinking about paying for an individual "off print" of a paper, may i encourage you to search elsewhere for that paper.
And for most of us who use ressearch papers, we don't pay for them individually: our libraries have licenses to the journals or the authors have made available "preprints" on their own websites.
There's a huge move in academic research towards "open access" (since most research in universities anyway is paid for through research funds, and if those are public funds the results should be publically available. Likewise there's a huge argument in terms of research impact that if one can't get at a paper digitally the paper might as well not exist. So academic authors find ways to make their papers available).
So what's going on here really?
Pilon's a smart guy. He does research, but he doesn't create peer reviewed papers himself, so he's not perhaps privy to the scene being described above. Fair enough. Even folks who do undergrad degrees and some masters degrees do not have to create peer reviewed research themselves. Consumption is not the same as production.
I'd suggest that, let's assume, the best intentions in the world, Pilon has still misrepresented scholarly practice, and in particular the role of the Scientific Abstract. Mayn't be his fault if this is how the outward facing journal products appear to a high consumer of such content. Maybe that's a problem for the scholarly community to take on board.
But really, Pilon's analogy of Abstract=Trailer is all noise and leger des mains around the main point he's trying to make: that the abstract is not the paper. His issue seems to be with folks who use abstracts to support their claims without having looked at the whole paper.
I'm not sure whom Pilon is thinking of, but for sure on online forums, lots of folks quote abstracts as support for their arguments without reading the paper. I'm not sure that's always a terrible thing, or leads to damaging conclusions. I'd have to know more about the intent of the exchange.
Likewise while, as Pilon suggests, one would not say they had seen the film because they'd seen the trailer, i don't know how many people actually lie about having read a paper when they've only read the abstract. Perhaps people act as if they've read the whole paper? The closest thing i've seen is the citation of abstracts as sufficient for supporting a claim. And sometimes they are. And sometimes seeing a trailer is absolutely sufficient to show it's either (a) it's not worth seeing or (b) all the best scenes were in the trailer. Now, i have to say, i have been surprised by the number of posts that have claimed that Avatar is like Dances with Wolves - i did not get that from either of the trailers i saw.
But if we assume there's more to the paper than the abstract, what is the difference, if the paper is just fleshing out the findings - the findings don't change.
Difference between Paper and Abstract: Here's what can change from reading the paper: by looking at the methodology of the paper - the how the study or experiment was actually done - one can begin to challenge the results. One might say - oh that population was not appropriate for generalizing the finding - that was done with non-trained participants rather than hard core athletes. I bet the findings would be different. OR you only used this muscle not that. That constrains the value of the finding.
One of my favorites is when someone states a result like a type of result was found with this approach (like longevity and fasting) - and then you find out the study is done with rats. And not that many of them either. Now in those cases that's not the abstract/paper's fault because all that's usually in the title - that's just how it can get reported.
So while reading the abstract is not the same as reading the paper, it's pretty durn good.
And while a professional researcher wouldn't get away with not reading the paper if they wanted to cite it as the basis for research, it is pretty durn legitimate to site a set of abstracts' results to show a trend. An informed reader or poster hopefully will start to get savvy enough to ask questions like "well what was the method" or "what was the population" or "how did they define X" to start to get at the important stuff to them.
Examples in Reading.Here's some examples of a blend of providing abstracts and some deeper readings of *certain* articles: b2d's series on DOMS is such a blend - requiring two parts - part 1 and part 2.
Maybe another good one is this survey on warm ups - that started with abstracts in order to ask the question where "warm up" is the term used in the abstract, but you have to read the paper to find out how a "warm up" is defined.
Apologies to Mr. Pilon. Anyway, as said, i dig Brad Pilon's work; he puts out loads of good stuff. I would be delighted to take the author out for coffee (and heck he lives in Canada. right on) or host him giving a talk or have a great discussion with him - so many of the folks i respect have been influenced by his work, and all for the good. This comment is in no way a criticism of him. And perhaps if i didn't actually teach this paper writing stuff and spend time working on abstracts with students, maybe i'd say ya sure abstract to movie trailer good comparison.
Except it's not. And not getting that may lead to furthering poor use and worse understanding of scientific practice - because the intents and also the designed uses are so different than a trailer that it's like - to use a food analogy - comparing apples to oranges.
And i do mean it: if this is what scientific practice appears in terms of public facing practice, we may have a job of work to do to address that. Tweet Follow @begin2dig
Abstract: Abstracts to scientific papers provide summaries of the question the study considered, the method used and the key findings from the work. Due to publishers access/licensing restrictions it is difficult for most people outside a formal research context, unless associated with a library that offers such access, to gain access to full papers of scholarly articles. What publishers do make available for free are abstracts of scholarly papers. While the intent of these abstracts is primarily for scholars to assess latest research findings and trends, and to make determinations about the need or not to read more of the full paper attached to an abstract, the general public doing an online search for information on a topic may find these abstracts. Without access to the full paper, they may do the best they can to make their case on the basis of a paper's findings alone. Recently, health/diet author Brad Pilon has taken issue with this abstract practice as potentially misrepresenting the actual work/paper. The following is a detailed consideration from the perspective scholarly authorship practice of the points Pilon makes, showing where there seem to be errors in Pilon's assertion of the intent or use of abstracts. The conclusion is that first, contrary to Pilon's assertions, abstracts are not designed to induce people to buy papers; they are designed for scholars to make research decisions and overview a field. Second, the use of abstracts in lieu of paper reading is not necessarily as problematic as Pilon seems to suggest. More would need to be know of the context for that assertion.
I have a lot of respect for the author of Eat Stop Eat. Brad Pilon has but together a lot of well reasoned arguments to for the possible benefits of intermittent ceases of eating (he wishes he hadn't called it fasting) for well being. Great. Today on facebook from a post by Richard Chignel i saw a link about Brad Pillon saying what a scientific abstract is. His thesis is that it's like a Movie Trailer - it's there to get you to buy the Paper. Speaking as a scientist who publishes scholarly papers, this is not my experience nor i suspect my colleagues. So i'd like to take this post to give you a perspective from the other side of the house, going through Pillon's points.
Abstracts Exaggerate? Beyond asserting that the abstract is like the trailer to get you to buy a paper, Pillon also makes some general claims about what's in an abstract. So first problem: a key error is to say an abstract "exagerates" findings in the paper. To generalize like that, a person would need to have a large number of examples. I'm stuck trying to think of one. And i'm a Reader - that's my job title. I read a lot of papers. I read even more abstracts. Exageration in abstracts is not a sustainable practice.
Journals - where scientific papers are published - are managed by editors. Editors of journals, by the way, are almost always also experienced and active researchers. So these editors are also writers. It's their job to make sure what's in the abstract is a fair reflection of the content/findings of the paper. If that were not the case, the journal would soon get a rep for being shoddy. You may say you don't know as a non-expert what journal is shunned or not. There are a bunch of metrics for that, but if you search for "journal rankings" in google, you'll see that there are publically available metrics. And a researcher from any space will make that kind of tour through journals a first port of call, to find out the standing of the publication.
Abstract Structure. This is what i teach our graduate students to consider when they're writing up their results. And if the seminar hadn't been snowed out today, it's what we would have been looking at formally. Let no good lecture go to waste - so let me share this here. The abstract needs at least the following ingredients:
- what is the problem being considered by the work reported?
- why is that problem important?
- what method was used to investigate the problem?
- what are the results?
It's about NOT reading the paper. The point of the abstract is to give one sufficient information to know if they need to BOTHER reading the paper for their area of work. THis may seem kinda odd. IF i've put time into writing this wonderful paper, don't i want everyone to read it? It's not the first thing that hits most researchers' minds.
The hope is that the work is sufficiently interesting or relevant that others will find it so (or find it at all, ever), and thus it will stand on its own merits when folks are looking for related work to their own in that area. These are specialized pieces of writing that assume a certain knowledge level for reading. They are written to advance understanding in a particular field. And their are tons produced annually. Hence people who look at research papers generally do so because they have a question they want to understand.
And when such a question comes up, they usually his a dedicated searchable index site like pubmed or other domain specific indexes - not google or not a journal publisher - to see lists of papers, and to click through to the abstracts, and from there, if desired, one of the links to the full text.
example of pubmed search results
So of this myriad of possiblely relevant information, the abstract acts as a filter.
Of the sometimes hundreds of papers available, which ones should be set aside to read further? That's a first pass. This may sound like that's a match for Pillon's try to get you to read the paper. But authors of this kind of work AREN't trying to get you to read a paper. They aren't trying to show you the best shots from the film. They aren't being highly selective about what they show you. There are those pretty standard, one might say objective conventions, about what has to be in an abstract. No jump cuts. The really dry straight stuff only. We're talking about a community here of people who get to know each other. If you as a scientist started trying to sex up your abstracts, you'd be meat.
And since this is a community practice, we all know that we want to make our abstracts as useful as possible so that another readers can make these determinations. Our own reputations are at stake here, too.
As an aside: another way researchers make determinations about what's important to read in an area: looking at the papers other credible researchers are looking at in that area.
Surveying the Field. The second pass is that the abstracts give a researcher a very good sense of what the trends have been in a particular topic over time. So the researcher can begin to get a sense of what questions have already been considered - very important if that person is planning their own work and doesn't want to reinvent the wheel.
Likewise it's cool to see when a particular topic trended. You can see when a particular question seemed to have been hot by how many papers related to it came out in a year. What happened that it trailed off? Is it now called something else? Is the problem solved?
It's not to get you to buy the paper. Of course there's a difference too between reading and buying a paper. Pillon quotes the price a publisher assigns to a PDF. That's the publisher. If anyone is thinking about paying for an individual "off print" of a paper, may i encourage you to search elsewhere for that paper.
And for most of us who use ressearch papers, we don't pay for them individually: our libraries have licenses to the journals or the authors have made available "preprints" on their own websites.
There's a huge move in academic research towards "open access" (since most research in universities anyway is paid for through research funds, and if those are public funds the results should be publically available. Likewise there's a huge argument in terms of research impact that if one can't get at a paper digitally the paper might as well not exist. So academic authors find ways to make their papers available).
So what's going on here really?
Pilon's a smart guy. He does research, but he doesn't create peer reviewed papers himself, so he's not perhaps privy to the scene being described above. Fair enough. Even folks who do undergrad degrees and some masters degrees do not have to create peer reviewed research themselves. Consumption is not the same as production.
I'd suggest that, let's assume, the best intentions in the world, Pilon has still misrepresented scholarly practice, and in particular the role of the Scientific Abstract. Mayn't be his fault if this is how the outward facing journal products appear to a high consumer of such content. Maybe that's a problem for the scholarly community to take on board.
But really, Pilon's analogy of Abstract=Trailer is all noise and leger des mains around the main point he's trying to make: that the abstract is not the paper. His issue seems to be with folks who use abstracts to support their claims without having looked at the whole paper.
I'm not sure whom Pilon is thinking of, but for sure on online forums, lots of folks quote abstracts as support for their arguments without reading the paper. I'm not sure that's always a terrible thing, or leads to damaging conclusions. I'd have to know more about the intent of the exchange.
Likewise while, as Pilon suggests, one would not say they had seen the film because they'd seen the trailer, i don't know how many people actually lie about having read a paper when they've only read the abstract. Perhaps people act as if they've read the whole paper? The closest thing i've seen is the citation of abstracts as sufficient for supporting a claim. And sometimes they are. And sometimes seeing a trailer is absolutely sufficient to show it's either (a) it's not worth seeing or (b) all the best scenes were in the trailer. Now, i have to say, i have been surprised by the number of posts that have claimed that Avatar is like Dances with Wolves - i did not get that from either of the trailers i saw.
But if we assume there's more to the paper than the abstract, what is the difference, if the paper is just fleshing out the findings - the findings don't change.
Difference between Paper and Abstract: Here's what can change from reading the paper: by looking at the methodology of the paper - the how the study or experiment was actually done - one can begin to challenge the results. One might say - oh that population was not appropriate for generalizing the finding - that was done with non-trained participants rather than hard core athletes. I bet the findings would be different. OR you only used this muscle not that. That constrains the value of the finding.
One of my favorites is when someone states a result like a type of result was found with this approach (like longevity and fasting) - and then you find out the study is done with rats. And not that many of them either. Now in those cases that's not the abstract/paper's fault because all that's usually in the title - that's just how it can get reported.
So while reading the abstract is not the same as reading the paper, it's pretty durn good.
And while a professional researcher wouldn't get away with not reading the paper if they wanted to cite it as the basis for research, it is pretty durn legitimate to site a set of abstracts' results to show a trend. An informed reader or poster hopefully will start to get savvy enough to ask questions like "well what was the method" or "what was the population" or "how did they define X" to start to get at the important stuff to them.
Examples in Reading.Here's some examples of a blend of providing abstracts and some deeper readings of *certain* articles: b2d's series on DOMS is such a blend - requiring two parts - part 1 and part 2.
Maybe another good one is this survey on warm ups - that started with abstracts in order to ask the question where "warm up" is the term used in the abstract, but you have to read the paper to find out how a "warm up" is defined.
Apologies to Mr. Pilon. Anyway, as said, i dig Brad Pilon's work; he puts out loads of good stuff. I would be delighted to take the author out for coffee (and heck he lives in Canada. right on) or host him giving a talk or have a great discussion with him - so many of the folks i respect have been influenced by his work, and all for the good. This comment is in no way a criticism of him. And perhaps if i didn't actually teach this paper writing stuff and spend time working on abstracts with students, maybe i'd say ya sure abstract to movie trailer good comparison.
Except it's not. And not getting that may lead to furthering poor use and worse understanding of scientific practice - because the intents and also the designed uses are so different than a trailer that it's like - to use a food analogy - comparing apples to oranges.
And i do mean it: if this is what scientific practice appears in terms of public facing practice, we may have a job of work to do to address that. Tweet Follow @begin2dig
PDFIf you enjoy these posts by all means you should
Subscribe to:
Post Comments (Atom)
COACHING with dr. m.c. 
1 comment:
Thanks mc. I really liked, and embraced Eat Stop Eat, and think Brad is a smart cookie, but this one stunk a bit to me.
It was almost as if he had a specific beef, or he is branding himself as the "journal conspiracy guy", who knows.
Post a Comment