Mrs. Lemming and I are fixing up the home study this month, which will entail painting, moving everything around, and intermittent computer access. I've queued up some auto posts to keep y'all amused. In the mean time, feel free to talk among yourselves, or answer the poll to the right.
Peer review is one of those exercises that is vitally important to the promulgation of scientific knowledge. It is also one for which we get absolutely no training. and if that is not bad enough, it is cloaked in unnecessary secrecy and anonymity. The way it works is this:
In between irate meetings, technical emergencies, slipping deadlines, and normal workload, you get an unsolicited email from a total stranger. Assuming it doesn’t go directly to the spam bin, opening it reveals that it is Professor Joe Blogger, associate editor of the Journal of Acceptable Results, and he’d like you to review a paper.
Hopefully the paper is on a topic with which you are actually familiar.
Generally there is a multiple choice card or sheet, in addition to the actual brain-required part of the job- namely writing intelligent and useful commentary on somebody else’s research.
As far as I can tell, there are two basic approaches. These are ideally similar but in practice can be opposite. The first is imitation of reviews that one has received. The second is the golden rule.
For me, the first option would involve some combination of non-specific pleasantries, attacks based on a misunderstanding of the literature or techniques used, or multi-page lists of specific minor points which are not related to each other or the using-whole-sentences part of the review. So when asked to review, I tried the second method, but without any real sense of guidance or idea what I was working towards. My inability to find a type specimen of a quality review is probably due to the lack of transparency that surrounds peer review. So in order to illuminate this mysterious process, here is my reviewing experience.
I reviewed my first paper a bit less than two years ago. The finished paper was finally published last southern winter while I was in the field, so I was unaware that it had come out until this week. My main aims were:
1. Pay close attention to the methods. In this case, the methodology used in the paper included techniques that I used, albeit with technical support and a long time ago. I asked: Are they appropriate? Will they overcome noise and interference problems? Do the authors demonstrate their awareness of what the pitfalls are, and show that they were avoided?
2. Look at the data. If it’s good, proceed. If not, double check it. If it’s better than the described methods is capable of producing, get very suspicious.
3. Is the result important to the study? If so, try to convey this to the editor, who is probably not an expert in both the experimental application AND the topic for which it is being applied. (He may not know much about either). I did not google the handling editor, but in hind sight I probably should have, in order to ascertain my audience.
4. Is it worth publishing? It is possible for a study to be correct, but unimportant. This point generally forms the central thesis of the review. All of the above points are used to justify it.
I generally don’t pay a huge amount of attention to conclusions or interpretations, as long as they are consistent with the results and with previous work. When reading papers professionally, I am generally data-mining or method-hunting, so it is hard to care about part of a paper that I never read. If the introduction is adequate, then the meaning of the results should be self-evident once the data is presented. Also, the way I see it, if a person does the hard yards and produces a good data set, he or she earns the right to arm-wave about it a little bit.
Note that what I paid attention to when evaluating the paper was not the same as what I wrote in the review. In particular, although I spent a long time looking at the methodology, I assumed the editor wasn't interested in knowing what the nitty-gritty details were. I assumed he just wanted to know if they were appropriate. As a result, I put most of my effort into describing why the results they present were significant. Unfortunately, I was not able to incorporate all suggestions into a coherent essay, and reverted to bullet point suggestions towards the end. If anyone has any tips or suggestions as to how to review effectively, please let me know in comments.
In this particular case, I did check the references, but I probably won’t in the future. That doesn’t require specialist knowledge, so anyone can do it.
So what does this all boil down to? Well, here's the final copy. It is posted with the surviving author’s knowledge and permission:
Dear Handling Editor,
The paper “Infrared and Raman spectroscopic observations on Central African carbonado and the implication to its origin” presents new data that constitutes a significant step forward in the study of this material.
While there have been several previous Raman studies of carbonado, none of them have looked at the possibility of annealing caused by polishing the stone, so that part of the paper is important. But the main value of this paper is the FTIR data.
Science has been trying to determine the IR spectra and nitrogen aggregation of carbonado diamond for almost 20 years. There have been numerous studies that have shown the important 1000-1400 cm-1 region is cluttered with interfering inclusions, and Garai et al. (2006) recently claimed to have an answer by simply interpreting what appear to be inclusions as diamond signal.
However this is the first paper to demonstrate the successful removal of interfering silicates, and to show that the remaining diamond spectra are consistent. The determination of N aggregation state for carbonado and the demonstration that fluid inclusions are intrinsic to the diamond are both key developments towards understanding the early history of this type of diamond.
There are a few minor improvements that should be made to the typescript before publication, however.
Firstly, the abstract and introduction should be read and edited by a native English speaker. There are several awkward sentence constructions that make the paper more difficult to read than it ought to be. The Raman conclusions are occasionally hard to understand as well.
The carbonado spectra in figures 4, 5, and 6 should be labeled so that we know which of the samples in table 1 the spectra correspond to. Figures 5 and 6 should show more than one carbonado spectrum, and figures 4 and 5 might be combinable into one figure.
A data table containing the Raman peak positions and widths observed should be included in the paper.
It would be nice if data files for the spectra were included in the supplementary material.
The Walker 1979 reference is not cited.
Overall, it is a good paper, and an important achievement in the field of carbonado study.
Sincerely,
Dr. Charles Magee
Professor David Karoly has an excellent post describing the relationship between climate and the recent Victorian bushfires.
Like everyone else on the internet, I recently watched the NASA Barriers to innovation and inclusion video produced by astronaut Andrew Thomas. It actually kept me up at night. “Process” is one of the main reasons that I’ve remained a technician/ project scientist ever since graduation- I have neither interest in nor a gift for bureaucratic fluff. I don’t even like verbosity. So one of the things I was thinking about last night was how fortunate I am to have a manager who basically throws himself under the bureaucratic train so that the rest of us scientists in the group can work relatively unhindered. He spent the day talking about process with suits; I changed a turbo pump in the morning and prepared analytical standards in lab all afternoon.
But as I was thinking about it, I realized that my luck wasn’t random chance. In part, it was natural selection.
I have terrible career skills. I’m terse in interviews, my resumé is plain, I don’t meta-analyse or talk in points. As a result, I am simply not hired by process-oriented employers, even when I do apply for such jobs. So the chances of me ending up in a situation like Heather’s are slimmer than they would be if I was an attractive candidate to such organizations.
I do intentionally pursue jobs that involve doing stuff, as opposed to arguing about what the best way to go about it is. But in addition to that I have this unconscious defense mechanism against red tape employment. This is good for me, but I wonder, how does it apply to the less privileged.
Now, I watched this video over at Sciencewomen’s blog. And one of their themes is how to improve the academic system to be more female friendly. But I wonder, could this be counterproductive?
Suppose the following:
1. Women and men are equally talented.
2. We remove systemic, but not personal barriers to women in academia.
3. Environments which favor systemic approaches stifle innovation.
If all three conditions are met, then women will be preferentially selected for jobs with stronger systemic structures, for the same reason that I don’t get those jobs. Assuming that these jobs stifle innovation, then it would be reasonable to expect these equally talented people to be less productive than equivalent folks in less structured environments. And the washed-up emeriti set could then use this productivity gap as evidence for an inherent gender-based lack of ability.
So I guess my point is that without individual change, systemic change is insufficient for producing equality of opportunity.
One of the big developments in geology over the past decade or so is to turn everything into a pretty picture. If it isn’t imaged, it’s mapped, and if it ain’t hyperspectral it needs to be saturated in coffee.
In one sense, this is just a continuation of a trend that started around 1800 with the first geologic map; it’s simply the best way to present complex spatial information. What makes the last decade or so different is simply the scale, speed, and breadth with which this has been applied.
For me, the turning point was EPMA element mapping. This is where the geologist goes out on the town, and while he or she is partying all night long, the electron probe maps out the elemental variation of a thin section. In the morning, the hangover is vanquished with a pretty picture showing such crazy things as REE zoning in optically homogenous garnets, or alteration features that render a year’s worth of work pointless.
In order to map things easily and rapidly, a few things are required. First, you need a fast analytical method. If you’re taking tens to hundreds of thousands of data points, you can’t spend all day on each one.
Secondly, you need to know where you are. You can’t make a map on any scale without positional information.
Thirdly, you need a computer to process the data so that all data is somehow normalized to a useful value, and all the bits are put together in the correct places.
Woodhead et al. show how all this can be done using laser ICPMS. The key for 1 is rapid washout time, and for 2 and 3 are computer-controlled stage operation. They did a bit of extra homework, though, and ran several experiments to show that ejecta resampling produced negligible signal contamination on the scale of their measurements. This is important with destructive mapping techniques, because the sample quality can be compromised by annihilating it with laser beams.
As their example experiment, they show some lovely stalactite images, with U-Mg-Sr spectral maps. You’ll have to read the paper- or guess- to learn which element the vertical axis represents.
Jon D. Woodhead, John Hellstrom, Janet M. Hergt, Alan Greig, Roland Maas (2007). Isotopic and Elemental Imaging of Geological Materials by Laser Ablation Inductively Coupled Plasma-Mass Spectrometry Geostandards and Geoanalytical Research, 31 (4), 331-343
Chris Clarke has put together a wonderful new blog carnival, focusing on those parts of the earth that get less than a foot of rain in a year. I highly recommend a trip to the Carnival of the Arid if you can't get to a real desert at the moment.
