I am one of those people, who is good at reading papers, but kind of mediocre at reviewing. That comes from the fact that I prefer giving and receiving inline and referenced comments Show information for the linked content (even if the span is as big as a section or multiple sections), rather than a general hand-wavy reviews I often tend to get from these conferences. I started reviewing as a final year undergrad, but I was never taught how to review. So, I slowly amassed all my information from Twitter, especially from Ahmad Beirami.
Some things to remember as a reviewer
03:07 PM · Aug 01, 2024 UTC If a paper clears the bar, give it a score ≥6.
Here is how I think about ratings:
- Should be oral? 8/9
- Should be spotlight? 7/8
- Clears the acceptance bar? 6/7
- Could be accepted after minor revs? 4/5
- Could be accepted after major revs? 3/4
- Fundamentally flawed 2/3
02:26 PM · Aug 05, 2024 UTC 5 leans more reject than accept so if you think a paper is good (with some minor revisions), then please give it 6+.
I reserve 5 for a good paper that needs non-trivial revisions that I'm uncomfortable to leave for camera ready which is rare.
In most cases scores are 6+ or 4-
03:07 PM · Aug 01, 2024 UTC If a paper clears the bar, give it a score ≥6.
Here is how I think about ratings:
- Should be oral? 8/9
- Should be spotlight? 7/8
- Clears the acceptance bar? 6/7
- Could be accepted after minor revs? 4/5
- Could be accepted after major revs? 3/4
- Fundamentally flawed 2/3
08:30 PM · Jul 31, 2024 UTC To the reviewer who claimed 8% improvement is marginal and not significant enough for a top conference paper:
The goal of a scientific paper is to further our collective understanding of how to solve problem, it's not to launch a new algorithm in production setting.
04:08 AM · Jan 23, 2024 UTC A periodic reminder to reviewers:
If you ask authors for more experiments, then you need to communicate a clear hypothesis you're trying to verify with those (e.g., effectiveness on imbalanced data, generalization beyond a certain modality, scalability, etc).
Otherwise don't!
Some QA
04:44 PM · Aug 05, 2024 UTC @abeirami What should be the score for a paper that clears the acceptance bar, in general, but uses a method that has many flaws unaccounted for -- but there are many published papers in the last 4-5 months that use the same method without accounting for those flaws?
06:27 PM · Aug 05, 2024 UTC @MimansaJ Depends on the nature of the flaws.
If the main claim of the paper is still valid but the evaluation is not extensive enough, I'd go with 6+ (and ask them to address remaining points in camera ready).
If the flaws might make the claims invalid, then I'd go with 4.
And some general comments
11:41 PM · Apr 12, 2024 UTC The review committee's job is to point out the flaws in a paper and give constructive feedback to improve the paper.
It's not to speculate how the flaws came about!
09:05 PM · Apr 12, 2024 UTC Just finished my ECCV reviews. 2 of the 6 papers in my pile were 100% unedited and incoherent LLM output. If you let ChatGPT write your paper, and one of your reviews is a "strong reject" and a diatribe about why what you did is immoral: hi, that was me, we are not friends.
02:44 AM · Aug 07, 2024 UTC If you decide to withdraw your paper without a rebuttal, it's nice to write a short (3-4 sentences) withdrawal note to thank the reviewers for their feedback, describe what you agree/disagree with, and what you plan to do.
Besides, you may get the same reviewers again.