skip to content
Site header image Mimansa Jaiswal

Collated Tips on Reviewing

This is a collection of posts by people, for people about reviewing in ML conferences interspersed with some of my own comments

Last Updated:
Academia Research

I am one of those people, who is good at reading papers, but kind of mediocre at reviewing. That comes from the fact that I prefer giving and receiving inline and referenced comments Show information for the linked content (even if the span is as big as a section or multiple sections), rather than a general hand-wavy reviews I often tend to get from these conferences. I started reviewing as a final year undergrad, but I was never taught how to review. So, I slowly amassed all my information from Twitter, especially from Ahmad Beirami.

Here are some of the amazing pieces of advice from his profile:

The question that a reviewer should ask themselves is: Does this paper take a gradient step in the right direction? Is the community better off with this paper published? If the answer is yes, then the recommendation should be to accept.
Ahmad Beirami
Ahmad Beirami
@abeirami
· Follow

If a paper clears the bar, give it a score ≥6. Here is how I think about ratings: - Should be oral? 8/9 - Should be spotlight? 7/8 - Clears the acceptance bar? 6/7 - Could be accepted after minor revs? 4/5 - Could be accepted after major revs? 3/4 - Fundamentally flawed 2/3

Ahmad Beirami
Ahmad Beirami
@abeirami

The question that a reviewer should ask themselves is: Does this paper take a gradient step in the right direction? Is the community better off with this paper published? If the answer is yes, then the recommendation should be to accept.

123
Reply
2 replies
If a paper clears the bar, give it a score ≥6. Here is how I think about ratings: - Should be oral? 8/9 - Should be spotlight? 7/8 - Clears the acceptance bar? 6/7 - Could be accepted after minor revs? 4/5 - Could be accepted after major revs? 3/4 - Fundamentally flawed 2/3
Ahmad Beirami
Ahmad Beirami
@abeirami
· Follow

5 leans more reject than accept so if you think a paper is good (with some minor revisions), then please give it 6+. I reserve 5 for a good paper that needs non-trivial revisions that I'm uncomfortable to leave for camera ready which is rare. In most cases scores are 6+ or 4-

Ahmad Beirami
Ahmad Beirami
@abeirami

If a paper clears the bar, give it a score ≥6. Here is how I think about ratings: - Should be oral? 8/9 - Should be spotlight? 7/8 - Clears the acceptance bar? 6/7 - Could be accepted after minor revs? 4/5 - Could be accepted after major revs? 3/4 - Fundamentally flawed 2/3

37
Reply
1 reply
5 leans more reject than accept so if you think a paper is good (with some minor revisions), then please give it 6+. I reserve 5 for a good paper that needs non-trivial revisions that I'm uncomfortable to leave for camera ready which is rare. In most cases scores are 6+ or 4-
What should be the score for a paper that clears the acceptance bar, in general, but uses a method that has many flaws unaccounted for--but there are many published papers in the last 4-5 months that use the same method without accounting for those flaws?
Depends on the nature of the flaws. If the main claim of the paper is still valid but the evaluation is not extensive enough, I'd go with 6+ (and ask them to address remaining points in camera ready). If the flaws might make the claims invalid, then I'd go with 4.
To the reviewer who claimed 8% improvement is marginal and not significant enough for a top conference paper: The goal of a scientific paper is to further our collective understanding of how to solve problems, it's not to launch a new algorithm in a production setting.
If you decide to withdraw your paper without a rebuttal, it's nice to write a short (3-4 sentences) withdrawal note to thank the reviewers for their feedback, describe what you agree/disagree with, and what you plan to do. Besides, you may get the same reviewers again.
The review committee's job is to point out the flaws in a paper and give constructive feedback to improve the paper. It's not to speculate how the flaws came about!
A periodic reminder to reviewers: If you ask authors for more experiments, then you need to communicate a clear hypothesis you're trying to verify with those (e.g., effectiveness on imbalanced data, generalization beyond a certain modality, scalability, etc). Otherwise don't!