I am one of those people, who is good at reading papers, but kind of mediocre at reviewing. That comes from the fact that I prefer giving and receiving inline and referenced comments Show information for the linked content (even if the span is as big as a section or multiple sections), rather than a general hand-wavy reviews I often tend to get from these conferences. I started reviewing as a final year undergrad, but I was never taught how to review. So, I slowly amassed all my information from Twitter, especially from Ahmad Beirami.
Here are some of the amazing pieces of advice from his profile:
If a paper clears the bar, give it a score ≥6. Here is how I think about ratings: - Should be oral? 8/9 - Should be spotlight? 7/8 - Clears the acceptance bar? 6/7 - Could be accepted after minor revs? 4/5 - Could be accepted after major revs? 3/4 - Fundamentally flawed 2/3
The question that a reviewer should ask themselves is: Does this paper take a gradient step in the right direction? Is the community better off with this paper published? If the answer is yes, then the recommendation should be to accept.
5 leans more reject than accept so if you think a paper is good (with some minor revisions), then please give it 6+. I reserve 5 for a good paper that needs non-trivial revisions that I'm uncomfortable to leave for camera ready which is rare. In most cases scores are 6+ or 4-
If a paper clears the bar, give it a score ≥6. Here is how I think about ratings: - Should be oral? 8/9 - Should be spotlight? 7/8 - Clears the acceptance bar? 6/7 - Could be accepted after minor revs? 4/5 - Could be accepted after major revs? 3/4 - Fundamentally flawed 2/3
What should be the score for a paper that clears the acceptance bar, in general, but uses a method that has many flaws unaccounted for -- but there are many published papers in the last 4-5 months that use the same method without accounting for those flaws?
Depends on the nature of the flaws. If the main claim of the paper is still valid but the evaluation is not extensive enough, I'd go with 6+ (and ask them to address remaining points in camera ready). If the flaws might make the claims invalid, then I'd go with 4.
The review committee's job is to point out the flaws in a paper and give constructive feedback to improve the paper. It's not to speculate how the flaws came about!
Just finished my ECCV reviews. 2 of the 6 papers in my pile were 100% unedited and incoherent LLM output. If you let ChatGPT write your paper, and one of your reviews is a "strong reject" and a diatribe about why what you did is immoral: hi, that was me, we are not friends.