Paper-thin quality

When I started doing research in artificial intelligence in games in 2001, I was one of the pioneers in this area. Such kind of research was seen by many as a bit ‘frivolous.’ I am glad to say that since then, this attitude has changed, and nowadays quite a few universities have scientists work on aspects of games. That is a good thing, because games comprise many elements that make them interesting for research. I might talk more about that in a future post, but in this post I want to make some statements about conferences in game research.

In 2001, there were very few conferences on game research. It was hard to find a place to submit papers to. Naturally, with the increasing popularity of game research, the number of conferences increased too. Nowadays there are several dozens of conferences and workshops in this area, and many more general conferences have a track on research in games. Personally, I am most interested in those conferences that focus exclusively on artificial intelligence in games, such as the AIIDE and the CIG.

Since I have contributed to quite a few of these conferences, I often get invited to act as referee. There was a time when I always accepted such an invitation, but nowadays I get so many of them that I sometimes have to refuse. This year I already reviewed about 30 papers for conferences, workshops, and journals. And I have noticed a disturbing trend.

The quality of the papers I get to review is decreasing rapidly. I am not entirely sure whether this is a general trend, or whether I have just been unlucky with the paper assignments, but I fear it is the first. Recently, I have been recommending rejections for about 80% of the papers that I get to review. And that is while I am a relatively ‘soft’ reviewer, who is usually willing to see if a paper is salvagable. Some of the grounds for rejection were:

  1. Submissions to conferences for which they are not suitable;
  2. Not being about scientific research;
  3. Excruciatingly bad English;
  4. Unoriginality and blandness;
  5. Vagueness in reporting; and
  6. Drawing conclusions that do not stand up to scrutiny.

I have a distinct feeling that many of the papers that I get to review are written by students. I have nothing against that in principle; I actually applaud having students write and present papers. But it is the job of students’ advisors to ensure that their papers are of acceptable quality. It seems to me that often advisors are keen to add their name to a paper but not assist in writing it.

Why do I get so many inferior papers to review? I think there are two main reasons.

The first reason is that game research is attractive, so many universities try to get on the bandwagon and do something in that area (nothing against that). Many universities also want to make a name for themselves, so they set up a conference or workshop around some theme in this area. But in getting papers they have to compete with all the other conferences and workshops that spring up out there. Some researchers have so many conferences and workshops where they can and want to send papers to, that they spread their research very thin, or write up very small results, or encourage their students to quickly write up their bachelor or master thesis results in paper format.

The second reason, and the most devastating one, is that the past has shown that many conferences indeed accept inferior papers. There are conferences out there that accept literally everything that is submitted to them. I have quite bad experiences with this. For instance, for a certain conference I refereed seven papers together with one other reviewer. We conversed about those papers and together rejected four of the seven. In the end we found that the conference organizers had simply accepted all seven of them.

Why do conferences do this? One reason is that they do not get a sufficient number of submissions, and to fill up their program they lower their standards. Another reason is that accepting papers boosts attendance: at most universities there is a policy that you only get to visit conferences where you have a paper accepted.

But what does it mean for me as a reviewer? Reviews take time, but they are a necessary part of doing science. Every scientist has to pay his dues in this respect. As a scientist, I do reviews to help out my fellow scientists, to boost research quality, and to get the occasional glimpse of interesting but yet unpublished research. What I expect from my fellow scientists is that they do their best to send in high-quality papers, with good research, grounded in science, focussed on an appropriate research area, and preferably written in acceptable English.

How do I respond when I get a paper that is unacceptable? If the authors attempted to do good research and write it down well, I do not mind writing a review that helps them to improve their work to acceptable levels. But if it is obvious that they just sent in a hastily thrown together piece of trash, then frankly, I feel insulted. I am expected to spend valuable time, which I could also spend on doing research, on reading and criticizing something that the authors themselves should have improved before submitting. They seem to say that their time spent on writing their paper is more valuable than my time spent on reviewing it.

All teachers know students who only start studying for exams after they failed one or more times. Evidently, such students aim to do the bare minimum needed for passing. I am wondering whether this attitude also persists with some scientists, whose goal it is to get published, no matter the quality of their work.

The main blame in this rests, in my opinion, with the conferences and workshops that set their standards too low. They encourage a bad attitude amongst authors. As a referee, my view is that they should simply not enlist my services if they intend to accept everything anyway. Yes, it sounds nice if your conference is ‘refereed,’ but if you accept everything, in practice it is not.

I think that, at present, we are in  shake-out phase. There are too many conferences and workshops in game research. Those that accept too much low-quality work will die out after a short while, simply because serious scientists do not want to visit them any longer. I think it is therefore in the best interests of conferences to keep their quality standards high. Quality over volume will persist in the end.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: