Center for Analysis and Design of Intelligent Agents

Reviewing Scientific Papers

Concepts

Conference	A gathering of scientists for the purpose of presenting their own work to each other.
Workshop	Conferences often have workshops on selected topics. These draw crowds from 8-40 people, depending on the size of the conference. Typically papers submitted to workshops are published in the conference proceedings.
Conference Proceedings	A publication of all the work presented at the conference, in the form of scientific papers.
Journal	A professional outlet/publication for scientific work.
Technical report	Anything can be published a technical report. Tech reports published by university departments are typically not peer-reviewed.
Measuring scientific prestige	Scientists compete. They compete for prestige, as measured by: 1. Number of publications. 2. Number of publications they are listed as first author on. 3. Number of publications per year. 4. Number of quotations by others to their work. 5. Quality of publications that papers get published in.
Prestige of scientific outlets	From low to high: * Tech report * Workshop paper * Conference paper * Book authored * Journal paper

Peer Review

What is it?	Before replication of results can be undertaken by the scientific community, results must be published. When a scientist reviews another scientist's current work, it's called “peer review”.
The peer	A scientist should be an authority in his/her field – is there anyone who has a higher authority? Yes, the scientific method, in other words the scientific community. To review their work current work scientists enlist the practical embodiment of this community – their peers.
How current scientific work gets evaluated	Via replication of results – but first results must be published. Deciding what gets published, and how, is the role of the peer review process.

The Peer Review Process

Step 0	Scientist does research, writes up results and submits a scientific paper to a selected outlet.
Step 1	Editor or conference chair receives submission, decides who should review. The selected review group, typically 3 or more scientists knowledgeable in the field in question, is called the peer review group.
Step 2	Paper sent to peer review group (typically 3 reviewers) with a deadline for returning their review, along with instructions.
Step 3	Editor gets reviews back from reviewers.
Step 4	Editor has to decide, based on reviews, whether to (1) accept paper as-is, with no changes (very rare!); (2) accept paper with minor revisions; (3) accept paper with major revisions; (4) reject paper.
Step 5	Editor sends result of reviews along with his decision for 1, 2, 3 or 4 above.
Step 6	Conclusion 1, great! You're done. Your paper will be published as-is. Conclusion 2: Use the reviews to improve your paper, send back to editor. Editor may request a shortlist of how you improved the paper. Your paper will be published with your changes. Conclusion 3: You will need to do major work to improve the paper (e.g. more experiments or compare more algorithms or systems). Your paper will probably be reviewed by the same 3 reviewers. The editor may ask you for a shortlist of how you addressed the reviewers' concerns.

Peer Review Instructions

Several categories are used when reviewing	Overall quality of the work; Novelty/Significance of the contributions to the field in question; Clarity of the writing; Language/writing quality; Adherence to guidelines (paper length, abstract length, etc.) for the publication in question.
Quality of Work	Use your experience with the subject, and of course with other papers. Look at the content, not where the author comes from or where he/she does the work. Be honest. Be fair. If it's bad, say so. You do not do anyone a favor by trying to “be nice” – in fact, being “nice” in this context translates to being cruel to all the readers, as well as the author (who will then not know that the quality is not good enough!). Always give reasons and examples to back up your comments.
Novelty/Significance	Sometimes papers are written that hardly make a dent in propelling science forward; sometimes papers are breakthrough. Here it is important to think about examples of work you know very well, and try to place this work in that context, to see where it fits in impact.
Clarity of writing	This is a prerequisite to getting a paper published. It doesn't matter how revolutionary the work is, if it's not communicated properly nobody will read it.
Language quality	Nothing is as annoying as a good paper that falls flat on bad use of English (which, let's face it, is the language of science, at least until China overpowers us with Mandarin). Be brutal! Do not be nice to badly written papers. NB: There is always room for improvement in this regard.

Being a Peer Reviewer

Rule number one	When you are requested to be a peer reviewer, accept. Firstly, it is your duty as a scientist to help out, even if it is not paid work! Secondly, it can really improve your own writing to read other people's papers!
When you get the paper + instructions	Note the deadline, then block out 1/2 day well before the deadline to do the review.
Follow the instructions	If you don't get any instructions, use the above that I gave you.
First mistake	Thinking it's “really easy”. Sometimes a paper is obviously flawed – but that is not the most common case. Most of the time it requires a lot of work, careful analysis and careful reading to see whether a paper is worthy of publication. Nevertheless, the bottom line is always that the author is ultimately responsible.
Second mistake	Thinking it's not so important. Your review could affect a fellow scientist's career!

Guidelines for Reviewing Papers

Typically you lay out the categories that matter in your review, even before you start.
Categories often used are:
- Clarity and ease of reading (including structure, figures, explanations, etc.)
- Quality of the written English (grammar, spelling, and related)
- Novelty - how much of an advance on current state of the art is the work (this should only play a minor role in your review here, since the assignment does not emphasize this factor)
- Impact - potential for the work to have impact, both scientific, technological, and business wise
- It is useful to have a category called “minor comments” or “other comments” where you put general points, spelling suggestions, questions about grammar, etc., because it is often easiest for the author to do a pass on these separately from deeper concerns about the content of the paper, structure, and other issues having to do with the content and that generally take much more time to fix.
Do at least two read-through passes - especially to ensure that your early comments are coherent and consistent with those made later (often you see e.g. a better place to make a comment than the initial place you mentioned it)
You should take notes while you read, some of which will probably change in a second pass
Keep these questions in mind at all times: What are the most important things for the author to address? What is the most useful way for me to explain what these issues are?

EOF