Review criteria¶
The JOSS paper¶
As outlined in the submission guidelines provided to authors, the JOSS paper (the compiled PDF associated with this submission) should only include:
- A list of the authors of the software
- Author affiliations
- A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience
- A clear statement of need that illustrates the purpose of the software
- Mentions (if applicable) of any ongoing research projects using the software or recent scholarly publications enabled by it
- A list of key references including a link to the software archive
Important
Note the paper should not include software documentation such as API (Application Programming Interface) functionality, as this should be outlined in the software documentation.
Review items¶
Important
Note, if you’ve not yet been involved in a JOSS review, you can see an example JOSS review checklist here.
Software license¶
There should be an OSI approved license included in the repository. Common licenses such as those listed on choosealicense.com are preferred. Note there should be an actual license file present in the repository not just a reference to the license.
Acceptable: A plain-text LICENSE file with the contents of an OSI approved license
Not acceptable: A phrase such as ‘MIT license’ in a README file
Documentation¶
There should be sufficient documentation for you, the reviewer to understand the core functionality of the software under review. A high-level overview of this documentation should be included in a README file (or equivalent). There should be:
A statement of need¶
The authors should clearly state what problems the software is designed to solve and who the target audience is.
Installation instructions¶
There should be a clearly-stated list of dependencies. Ideally these should be handled with an automated package management solution.
Good: A package management file such as a Gemfile or package.json or equivalent
OK: A list of dependencies to install
Bad (not acceptable): Reliance on other software not listed by the authors
Example usage¶
The authors should include examples of how to use the software (ideally to solve real-world analysis problems).
API documentation¶
Reviewers should check that the software API is documented to a suitable level.
Good: All functions/methods are documented including example inputs and outputs
OK: Core API functionality is documented
Bad (not acceptable): API is undocumented
Note
The decision on API documentation is left largely to the discretion of the reviewer and their experience of evaluating the software.
Community guidelines¶
There should be clear guidelines for third-parties wishing to:
- Contribute to the software
- Report issues or problems with the software
- Seek support
Functionality¶
Reviewers are expected to install the software they are reviewing and to verify the core functionality of the software.
Tests¶
Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.
Good: An automated test suite hooked up to an external service such as Travis-CI or similar
OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g., a sample input file to assert behavior)
Bad (not acceptable): No way for you the reviewer to objectively assess whether the software works
Other considerations¶
Authorship¶
As part of the review process, you are asked to check whether the submitting author has made a ‘substantial contribution’ to the submitted software (as determined by the commit history) and to check that ‘the full list of paper authors seems appropriate and complete?’
As discussed in the submission guidelines for authors, authorship is a complex topic with different practices in different communities. Ultimately, the authors themselves are responsible for deciding which contributions are sufficient for co-authorship, although JOSS policy is that purely financial contributions are not considered sufficient. Your job as a reviewer is to check that the list of authors appears reasonable, and if it’s not obviously complete/correct, to raise this as a question during the review.
An important note about ‘novel’ software and citations of relevant work¶
Submissions that implement solutions already solved in other software packages are accepted into JOSS provided that they meet the criteria listed above and cite prior similar work. Reviewers should point out relevant published work which is not yet cited.
What happens if the software I’m reviewing doesn’t meet the JOSS criteria?¶
We ask that reviewers grade submissions in one of three categories: 1) Accept 2) Minor Revisions 3) Major Revisions. Unlike some journals we do not reject outright submissions requiring major revisions - we’re more than happy to give the author as long as they need to make these modifications/improvements.
What about submissions that rely upon proprietary languages/development environments?¶
As outlined in our author guidelines, submissions that rely upon a proprietary/closed source language or development environment are acceptable provided that they meet the other submission requirements and that you, the reviewer, are able to install the software & verify the functionality of the submission as required by our reviewer guidelines.
If an open source or free variant of the programming language exists, feel free to encourage the submitting author to consider making their software compatible with the open source/free variant.