10 big questions to dissect science papers

With a new semester under way, serious science students will have to master the skill of dissecting scientific journal articles – coded in mysterious titles, impenetrable language, reams of data, and numerous authors.

At least one group of undergraduate and master’s degree students should have a leg up on their peers – 26 students from around the world who attended the 10-week Novartis Summer Scholars program in Cambridge, MA, and Emeryville, CA.

Novartis recruits promising young scientists--most from historically underserved communities and some from as far away as Nepal and Zimbabwe--who want to improve their chances to attain advanced science degrees. In addition to hands-on lab instruction from Novartis scientists, the students attend career seminars and science workshops.

Dr. M. William Lensch,
Harvard Stem Cell Institute

How to read a scientific paper was one of the standout lectures in Cambridge. Speaker M. William Lensch, Ph.D., Executive Director of the Harvard Department of Stem Cell and Regenerative Biology and Faculty Director of Education for the Harvard Stem Cell Institute, said that successful papers rest on an organized structure that formulates a hypothesis, makes predictions, and has experiments, which are repeated as needed. The authors’ “spin” on the paper, their interpretation of the data, is provided in the abstract, as well as in the more lengthy “discussion” section at the end, but the devil is in the details sprinkled throughout.

Yet, Lensch has found, that less experienced scientists are more likely to rely on the author’s interpretations rather than drawing their own conclusions from their own review of the data. He urged students to take a deeper dive by asking the following 10 main questions:

1. What is the central point the authors are trying to prove? Read carefully to determine whether the authors actually get there or fall short and/or over-interpret results.
2. Does it matter how often an experiment is repeated? Generally if you get a specific result once, it could be a mistake. If you get the same result twice, it could be a coincidence. If you get the same result three times, then maybe you have an interesting story.
3. Are there any controls? An experiment is usually only as good as its controls, which should address multiple aspects of the experiment including sensitivity of assays used, what a “positive” result should look like, and what a “negative” result should look like. This allows an experiment to be evaluated by comparing experimental findings to multiple standards.
4. Do they use the best methods available? This brings up several related questions. Check whether they used methods that are qualitative (i.e., is something there or not) versus quantitative (i.e., if it’s there, how abundant is it?). Consider whether they use rigorous and complementary approaches such as cellular lineage-tracing, clonal analysis, and/or functional assays. Do they overly rely on a simple phenotype (i.e., something sort of “looks” like “X,” so they call it “X”)?
   
5. Always ask yourself, “What experiment didn’t they do that could make the data more convincing?” For example, if a paper shows that a given cell expresses muscle-specific genes that look like muscle cells under the microscope, and that when transplanted into a mouse’s damaged muscle integrate and survive, what additional experiment would better convince you that they are in fact bona fide muscle?
6. Did they perform enrichment (i.e., it is mostly this) vs. purification (i.e., it is only this) vs. no idea what they used? If they knock-out/down a gene, is there also a knock-out model? Do they agree or disagree? One would hope that they would produce similar effects. If so, what phenotype does it have? Is the result in an animal model the same as for the cognate human condition? If no, it may actually not be a very useful model of the human disease.
7. Did they increase confidence in the outcome by proving their results in multiple ways? Did they take the time to try to disprove their findings?
8. Is the paper “descriptive/hypothesis generating” or is it “hypothesis-driven” with multiple examples to back up the bottom line?
9. What do other articles say about the topic? If the paper is far afield from what others have published, are its big claims backed up by convincing data? How good is the bibliography? A good research paper should not cite too many review or opinion papers over peer-reviewed, primary research articles from high-impact journals.
10. Have the data been reproduced by other labs? Reproducibility is a key feature of good science and with a brand new finding, only time will tell if others are able to get it to work the same way.

With these questions, the Novartis students were given valuable information that could help them keep up to speed with the latest news in their fields and make meaningful contributions to peer groups as they compete for top spots in the best graduate programs.

Scientific Summer Scholars Program information