#ICML2025 outstanding position paper: Interview with Jaeho Kim on addressing the problems with conference reviewing

At this year’s International Conference on Machine Learning (ICML2025), Jaeho Kim, Yunseok Lee and Seulki Lee won an outstanding position paper award for their jobPosition: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards We learn through Jaeho regarding the troubles they were attempting to deal with, and their suggested writer responses system and customer incentive system.

Could you claim something regarding the issue that you deal with in your manifesto?

Our manifesto attends to the troubles tormenting present AI seminar peer evaluation systems, while likewise questioning regarding the future instructions of peer evaluation.

The unavoidable issue with the present peer evaluation system in AI seminars is the rapid development in paper entries driven by raising rate of interest in AI. To place this with numbers, NeurIPS obtained over 30,000 entries this year, while ICLR saw a 59.8% rise in entries in simply one year. This massive rise in entries has actually developed an essential inequality: while paper entries expand greatly, the swimming pool of certified customers has actually not kept up.

Entries to several of the significant AI seminars over the previous couple of years.

This inequality has extreme repercussions. Most of documents are no more getting sufficient evaluation high quality, weakening peer evaluation’s necessary feature as a gatekeeper of clinical understanding. When the evaluation procedure stops working, unsuitable documents and problematic study can slide via, possibly contaminating the clinical document.

Taking into consideration AI’s extensive social effect, this malfunction in quality assurance postures threats that prolong much past academic community. Poor study that gets in the clinical discussion can deceive future job, affect plan choices, and inevitably prevent real understanding improvement. Our manifesto concentrates on this important inquiry and suggests techniques on just how we can boost the high quality of evaluation, therefore resulting in much better circulation of understanding.

What do you suggest for in the manifesto?

Our manifesto suggests 2 significant modifications to deal with the present peer evaluation dilemma: a writer responses system and a customer incentive system.

Initially, the writer responses system makes it possible for writers to officially review the high quality of evaluations they obtain. This system permits writers to examine customers’ understanding of their job, recognize possible indications of LLM-generated material, and develop fundamental safeguards versus unjust, prejudiced, or surface evaluations. Notably, this isn’t regarding punishing customers, however instead producing marginal liability to safeguard writers from the little minority of customers that might not satisfy expert criteria.

2nd, our customer reward system offers both instant and lasting expert worth for high quality examining. For temporary inspiration, writer assessment ratings identify qualification for electronic badges (such as “Leading 10% Customer” acknowledgment) that can be shown on scholastic accounts like OpenReview and Google Scholar. For lasting job effect, we suggest unique metrics like a “customer effect rating”– basically an h-index computed from the succeeding citations of documents a customer has actually reviewed. This deals with customers as factors to the documents they aid enhance and verifies their duty ahead of time clinical understanding.

Could you inform us even more regarding your proposition for this brand-new two-way peer evaluation approach?

Our suggested two-way peer evaluation system makes one essential adjustment to the present procedure: we divided evaluation launch right into 2 stages.

The writers’ suggested alteration to the peer-review system.

Presently, writers send documents, customers create total evaluations, and all evaluations are launched at the same time. In our system, writers very first obtain just the neutral areas– the recap, toughness, and concerns regarding their paper. Writers after that give responses on whether customers correctly recognized their job. Just hereafter responses do we launch the 2nd component having weak points and rankings.

This technique provides 3 primary advantages. Initially, it’s sensible– we do not require to alter existing timelines or evaluation design templates. The 2nd stage can be launched quickly after the writers provide responses. Second, it shields writers from careless evaluations considering that customers understand their job will certainly be reviewed. Third, considering that customers normally evaluate numerous documents, we can track their responses ratings to aid location chairs recognize (ir) liable customers.

The essential understanding is that writers understand their very own job best and can promptly detect when a customer hasn’t correctly involved with their paper.

Could you speak about the concrete incentive system that you recommend in the paper?

We suggest both temporary and lasting incentives to deal with customer inspiration, which normally decreases gradually regardless of beginning enthusiastically.

Temporary: Digital badges presented on customers’ scholastic accounts, granted based upon writer responses ratings. The objective is making customer payments a lot more noticeable. While some seminars listing leading customers on their sites, these listings are tough to locate. Our badges would certainly be plainly shown on accounts and can also be published on seminar name tags.
Instance of a badge that can show up on accounts.

Long-lasting: Mathematical metrics to measure customer effect at AI seminars. We recommend tracking steps like an h-index for assessed documents. These metrics can be consisted of in scholastic profiles, comparable to just how we presently track magazine effect.

The core concept is producing concrete job advantages for customers while developing peer evaluation as an expert scholastic solution that compensates both writers and customers.

What do you believe could be several of the advantages and disadvantages of executing this system?

The advantages of our system are threefold. Initially, it is a really sensible remedy. Our technique does not alter present evaluation routines or evaluation worries, making it very easy to include right into existing systems. Second, it motivates customers to act even more properly, recognizing their job will certainly be reviewed. We stress that the majority of customers currently act properly– nonetheless, also a handful of careless customers can seriously harm the peer evaluation system. Third, with enough range, writer responses ratings will certainly make seminars a lot more lasting. Location chairs will certainly have much better info regarding customer high quality, allowing them to make even more educated choices regarding paper approval.

Nonetheless, there is solid capacity for video gaming by customers. Customers could maximize for incentives by offering extremely favorable evaluations. Actions to combat these troubles are certainly required. We are presently discovering services to resolve this concern.

Exist any kind of ending ideas you want to include regarding the possible future
of seminars and peer-review?

One arising fad we have actually observed is the raising conversation of LLMs in peer evaluation. While our team believe present LLMs have numerous weak points (e.g., punctual shot, superficial evaluations), we likewise believe they will ultimately exceed human beings. When that takes place, we will deal with an essential issue: if LLMs give much better evaluations, why should human beings be examining? Equally as the fast surge of LLMs captured us not really prepared and wreaked havoc, we can not manage a repeat. We must begin planning for this inquiry asap.

Concerning Jaeho

Jaeho Kim is a Postdoctoral Scientist at Korea College with Teacher Changhee Lee. He obtained his Ph.D. from UNIST under the guidance of Teacher Seulki Lee. His primary study concentrates on time collection discovering, especially establishing structure designs that produce artificial and human-guided time collection information to decrease computational and information prices. He likewise adds to enhancing the peer evaluation procedure at significant AI seminars, with his job identified by the ICML 2025 Exceptional Statement Of Principles Honor.

Check out the operate in complete

Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards, Jaeho Kim, Yunseok Lee, Seulki Lee