--- title: "Collated Tips on Reviewing" slug: collated-tips-on-reviewing canonical_url: https://mimansajaiswal.github.io/posts/collated-tips-on-reviewing/ collection: Fieldnotes published_at: 2024-08-07T00:00:00.000Z updated_at: 2025-09-11T00:00:00.000Z tags: - Academia - Research excerpt: "This is a collection of posts by people, for people about reviewing in ML conferences interspersed with some of my own comments" author: "Mimansa Jaiswal" --- ## Navigation Context - Canonical URL: https://mimansajaiswal.github.io/posts/collated-tips-on-reviewing/ - You are here: Home > Posts > Fieldnotes > Collated Tips on Reviewing ### Useful Next Links - [Home](https://mimansajaiswal.github.io/) - [Publications](https://mimansajaiswal.github.io/papers/) - [Blurbs](https://mimansajaiswal.github.io/collections/blurbs/) - [Fieldnotes](https://mimansajaiswal.github.io/collections/fieldnotes/) - [Personal](https://mimansajaiswal.github.io/collections/personal/) - [Research](https://mimansajaiswal.github.io/collections/research/) ### Related Content #### Other Pages Mentioned On This Page - [Please Don’t Have 10 Textareas in Review Forms](https://mimansajaiswal.github.io/posts/please-dont-have-10-textareas-in-review-forms/) I am one of those people, who is good at reading papers, but kind of mediocre at reviewing. That comes from the fact that I prefer giving and receiving inline and referenced comments (even if the span is as big as a section or multiple sections), rather than a general hand-wavy reviews I often tend to get from these conferences. I started reviewing as a final year undergrad, but I was never taught how to review. So, I slowly amassed all my information from Twitter, especially from [Ahmad Beirami](https://x.com/abeirami). **Some things to remember as a reviewer** ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1514248058619379713) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `02:24 PM · Apr 13, 2022 UTC` The question that a reviewer should ask themselves is: Does this paper take a gradient step in the right direction? Is the community better off with this paper published? If the answer is yes, then the recommendation should be to accept. [💬](https://x.com/intent/tweet?in_reply_to=1514248058619379713) [❤️](https://x.com/intent/like?tweet_id=1514248058619379713) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1819027168476713327) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `03:07 PM · Aug 01, 2024 UTC` If a paper clears the bar, give it a score ≥6. Here is how I think about ratings: \- Should be oral? 8/9 \- Should be spotlight? 7/8 \- Clears the acceptance bar? 6/7 \- Could be accepted after minor revs? 4/5 \- Could be accepted after major revs? 3/4 \- Fundamentally flawed 2/3 [💬](https://x.com/intent/tweet?in_reply_to=1819027168476713327) [❤️](https://x.com/intent/like?tweet_id=1819027168476713327) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1514248058619379713) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `02:24 PM · Apr 13, 2022 UTC` The question that a reviewer should ask themselves is: Does this paper take a gradient step in the right direction? Is the community better off with this paper published? If the answer is yes, then the recommendation should be to accept. [💬](https://x.com/intent/tweet?in_reply_to=1514248058619379713) [❤️](https://x.com/intent/like?tweet_id=1514248058619379713) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1820466511536099651) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `02:26 PM · Aug 05, 2024 UTC` 5 leans more reject than accept so if you think a paper is good (with some minor revisions), then please give it 6+. I reserve 5 for a good paper that needs non-trivial revisions that I'm uncomfortable to leave for camera ready which is rare. In most cases scores are 6+ or 4- [💬](https://x.com/intent/tweet?in_reply_to=1820466511536099651) [❤️](https://x.com/intent/like?tweet_id=1820466511536099651) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1819027168476713327) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `03:07 PM · Aug 01, 2024 UTC` If a paper clears the bar, give it a score ≥6. Here is how I think about ratings: \- Should be oral? 8/9 \- Should be spotlight? 7/8 \- Clears the acceptance bar? 6/7 \- Could be accepted after minor revs? 4/5 \- Could be accepted after major revs? 3/4 \- Fundamentally flawed 2/3 [💬](https://x.com/intent/tweet?in_reply_to=1819027168476713327) [❤️](https://x.com/intent/like?tweet_id=1819027168476713327) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1818746095289483292) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `08:30 PM · Jul 31, 2024 UTC` To the reviewer who claimed 8% improvement is marginal and not significant enough for a top conference paper: The goal of a scientific paper is to further our collective understanding of how to solve problem, it's not to launch a new algorithm in production setting. [💬](https://x.com/intent/tweet?in_reply_to=1818746095289483292) [❤️](https://x.com/intent/like?tweet_id=1818746095289483292) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1749645234953138422) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `04:08 AM · Jan 23, 2024 UTC` A periodic reminder to reviewers: If you ask authors for more experiments, then you need to communicate a clear hypothesis you're trying to verify with those (e.g., effectiveness on imbalanced data, generalization beyond a certain modality, scalability, etc). Otherwise don't! [💬](https://x.com/intent/tweet?in_reply_to=1749645234953138422) [❤️](https://x.com/intent/like?tweet_id=1749645234953138422) **Some QA** ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/MimansaJ/status/1820501242696831456) Mimansa Jaiswal · [@MimansaJ](https://x.com/MimansaJ) · `04:44 PM · Aug 05, 2024 UTC` [@abeirami](https://x.com/abeirami) What should be the score for a paper that clears the acceptance bar, in general, but uses a method that has many flaws unaccounted for -- but there are many published papers in the last 4-5 months that use the same method without accounting for those flaws? [💬](https://x.com/intent/tweet?in_reply_to=1820501242696831456) [❤️](https://x.com/intent/like?tweet_id=1820501242696831456) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1820526950764720573) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `06:27 PM · Aug 05, 2024 UTC` [@MimansaJ](https://x.com/MimansaJ) Depends on the nature of the flaws. If the main claim of the paper is still valid but the evaluation is not extensive enough, I'd go with 6+ (and ask them to address remaining points in camera ready). If the flaws might make the claims invalid, then I'd go with 4. [💬](https://x.com/intent/tweet?in_reply_to=1820526950764720573) [❤️](https://x.com/intent/like?tweet_id=1820526950764720573) **And some general comments** ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1778931486600634746) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `11:41 PM · Apr 12, 2024 UTC` The review committee's job is to point out the flaws in a paper and give constructive feedback to improve the paper. It's not to speculate how the flaws came about! [💬](https://x.com/intent/tweet?in_reply_to=1778931486600634746) [❤️](https://x.com/intent/like?tweet_id=1778931486600634746) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/jon_barron/status/1778892170340249657) Jon Barron · [@jon\_barron](https://x.com/jon_barron) · `09:05 PM · Apr 12, 2024 UTC` Just finished my ECCV reviews. 2 of the 6 papers in my pile were 100% unedited and incoherent LLM output. If you let ChatGPT write your paper, and one of your reviews is a "strong reject" and a diatribe about why what you did is immoral: hi, that was me, we are not friends. [💬](https://x.com/intent/tweet?in_reply_to=1778892170340249657) [❤️](https://x.com/intent/like?tweet_id=1778892170340249657) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b7/X_logo.jpg/240px-X_logo.jpg) [🔗](https://x.com/abeirami/status/1821014424255107213) Ahmad Beirami · [@abeirami](https://x.com/abeirami) · `02:44 AM · Aug 07, 2024 UTC` If you decide to withdraw your paper without a rebuttal, it's nice to write a short (3-4 sentences) withdrawal note to thank the reviewers for their feedback, describe what you agree/disagree with, and what you plan to do. Besides, you may get the same reviewers again. [💬](https://x.com/intent/tweet?in_reply_to=1821014424255107213) [❤️](https://x.com/intent/like?tweet_id=1821014424255107213) * * * ## Cite This Page ``` @article{jaiswal2024collatedtipsonr, title = {Collated Tips on Reviewing}, author = {Jaiswal, Mimansa}, journal = {mimansajaiswal.github.io}, year = {2024}, month = {Aug}, url = {https://mimansajaiswal.github.io/posts/collated-tips-on-reviewing/} } ```