Research Paper Reviews

Posted: December 8, 2007 in Thoughts
Tags: , , , ,

I just found this interesting page, whose author criticizes a research paper that has appeared in VLDB 2006 (a major Database research conference). The author of that page (not the paper) proceeds to point out some major (and minor) flaws in that paper, and also tries to find reasons why this paper did not get rejected. Here is an interesting extract from that page:

There is a common belief that the conference program committees never read more than a few pages of a conference paper when they make a decision on whether to accept or reject a paper. Since the statement contradicting the claim in the abstract is near the end of the tenth page, it would be easy to for such a referee to miss the contradiction.

The author also provides some suggestions on how to improve the review process. Some of the suggestions are quite obvious, like “authors should read their own papers”, while other suggestions might be a bit more drastic and need some time to see whether or not they make sense, like having one reviewer responsible for the paper, and getting their name associated with the paper if it is accepted (hmmm, what about blind reviews?)

It would be interesting to see if those suggestion could be implemented or not, and why. Any comments?

  1. Nosayba says:

    Your post reminded me of a paper that I’ve come across that implements some evolutionary computation algorithm. The charts in the Experimental results section were disastrous.. The ironic part is that the authors actually mention in some paragraph in that section that the results they got displayed an awkward behavior that they intend to improve in “Future work”! (that’s the beauty of the Future Work section).

    Some researchers submit papers that may be missing some parts or need more work to be done, for the sake of getting useful feedback from the Conference judges. Actually, I’ve done that once, and although the paper was rejected but the feedback was very beneficial. In some cases though, the paper does get accepted due to improper review like the one described in the link you referenced (Which btw gives Access Denied message). Maybe a good solution is to increase the number of judges, you know, get more opinions and see what the majority ends up with.

    Thank you for the implied, in-direct tips. In my next paper, I will focus on the first few pages only 🙂

  2. Amr El-Helw says:

    The way I see it, the problem is not just in the “unworthy” papers that get accepted. The real problem lies in the very good papers that get rejected. I’m not talking about bad papers that deserve to be rejected, but really strong papers that get rejected for the wrong reasons, including (but not limited to) the following:
    1. Could be due to the fact that the reviewer(s) did not actually take the time to read and understand the paper.
    2. Since other “bad” papers might have gotten accepted by mistake, thus taking a spot which could have been left to a better paper.

    I guess this all means that the reviewers have to put in a little more effort to be as fair as possible.

    By the way, I guess the page I pointed to has been removed, I couldn’t find it again 😦

  3. Nosayba says:

    Yes and,
    3. Language flaws for non native English speakers.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s