As use of artificial intelligence (AI) tools and large language models (LLMs) such as Claude and ChatGPT is increasing, the AAS journals offer the following guidance for the use of these tools while refereeing.
1. Role of the referee
Peer review is requested because we, as a community, are best placed to judge the quality and content of material in our own journals. Referees are therefore required to carefully read the submitted paper in its entirety, and they must be able to make a well-justified recommendation to the editor. No tool should be used in a way that avoids these essential parts of the referee’s job.
Referees are also reminded that the manuscript is shared with them in confidence, and that they are responsible for the protection of the authors’ intellectual property while carrying out a review. No use of an LLM should ever compromise this.
2. Model security and copyright
Referees are asked to keep the contents of the submission they are considering confidential, and — even in cases where the paper is on arXiv — the discussion with the authors and editors remains confidential. Therefore, referees may not use — in any aspect of the refereeing process — any LLM that might use uploaded material for training. Referees should carefully check the licensing of any installed system before using them.
Referees who do use LLMs should be aware that models may be out of date, or equally may be influenced by prior versions of the submitted or any other work on arXiv or elsewhere online. Care should be taken to make sure that the review concerns the work as submitted to the journal, rather than any other version.
3. Transparency in use
In keeping with the existing guidelines for authors, referees should declare where LLMs have been used and for what purpose. Examples of acceptable use might include improvements to the language in the referee report or a review of source code associated with a submission. Examples of unacceptable use might include summarizing the paper without the referee reading all of the text themselves. Where suggestions for improvement are made directly by an LLM, they should be identified as such in the report.
4. Accuracy and style
The referee retains responsibility for accuracy as the sole author of the submitted referee report and should carefully check any generated comments and references. Current LLMs are likely to make very generic suggestions for improvement (e.g., “Authors should consider carefully the use of statistical tests, and consider alternatives.”). Referees should ensure that all requests are reasonable, and that they make sense in the context of the paper being considered.
5. Author responses
The requirement for transparency also applies to authors, who should declare any use of an LLM in preparing a response to the editor or the referee. It may make sense to divide responses into assisted and unassisted sections.
