Procedure for Artifact Processing
This guide-line for conference chairs and authors explains the artifacts-publishing process with ACM,
written from the viewpoint of Conference Publishing Consulting (ConfPub).
This procedure starts around the date when the artifact-evaluation chairs collect artifacts.
There are four important deadlines:
the artifact-submission deadline (author's action),
the notification deadline (chairs' action),
the artifact-publication deadline (authors' action), and
the delivery of final meta data, artifacts, and proceedings to ACM (our action).
Requirements for Artifact Archiving
- Identification: Using DOIs to identify published objects is standard.
It is important to use a DOI that points to the specific version with which
the results of the paper can be reproduced
(for Zenodo: do not use the "always latest" DOI;
for FigShare: use a DOI with a version suffix, e.g., ".v1").
- Long-Term Availability: It is necessary that the artifacts are archived in an
archive that hosts the artifacts on a long-term basis, such as in digital libraries
of the ACM, Zenodo, etc.
(version repositories do not fulfill this requirement, as the hosting company
could decide at any point in time to discontinue the service,
as done by Google, for example: Google Code).
- Immutability: It is necessary that the artifact cannot be changed after publication,
because the reader needs to use the material exactly as the authors did to obtain their result.
Evaluation Categories and Badges
For ACM publications, the ACM Policy on Artifact Review and Badging applies.
There are three categories in the process:
- Availability of Artifacts (green badge): The artifact is available for others to validate the results in the article.
The hosting digital library provides a DOI to identify and cite the artifact,
ensures long-term availability and immutability.
This badge is independent from the evaluation process.
- Evaluation of Artifacts (red badges): The artifact was evaluated by an artifact-evaluation committee.
There are two badges "Functional" and "Reusable" that can be assigned.
The conditions for the "Reusable" badge imply the conditions for the "Functional" badge,
therefore, only one of the two badges can be assigned to a article.
- Evaluation of Results in Article (blue badges): The results of the article were validated by researchers
that are independent from the authors.
The badge "Results Validated --- Replicated" is assigned if the experiments from the article
were successfully replicated, and
the badge "Results Validated --- Reproduced" is assigned if the results from the article
were successfully reproduced using the author-provided artifact.
Section "Data-Availability Statement"
It is strongly recommended that
a section in the paper (before the references) describes the availability of data and software.
(For grant proposals, it is already a standard practice to provide such a declaration in many countries.)
The artifact should be included in the references section and cited just like other archived literature.
Note that this should be done before the camera-ready deadline, because the Availability badge
is assigned independently from the artifact-evaluation process (and its outcome).
The section in the paper should describe what should be used for reproduction
(pointing to a specific version of an archive, e.g., at Zenodo) and
what for reuse (a URL of, e.g., a GitLab or project home page).
The badges for the evaluation results ("Functional", "Reusable", "Reproduced") will be added to the paper
automatically during the paper processing.
If data cannot be made available, this section should explain how the data and software necessary to repeat the experiments can be obtained.
For some conferences, this section is treated like references and not counted against the page limit.
How are Artifacts Represented in the ACM DL?
For each artifact, we deliver meta data to ACM, such that each artifact has an own landing page in the ACM DL.
For example: The paper doi:10.1145/3540250.3549172
has a section "Related Artifact" on the landing page for the paper,
and the corresponding artifact doi:10.5281/zenodo.7082407
also has an own landing page in the ACM DL, which points to Zenodo as well as back to the paper.
This way, artifacts are treated as first-class bibliographic objects and are easy to find.
Tasks and Responsibilities (Artifact-Evaluation Chair <---> ConfPub)
-
1. Chair collects artifacts for review.
It is important to make sure to use the very same artifact that the authors used for their article.
Chairs have the following options to identify the artifacts (non-exclusive list)):
- DOI: authors have published the artifact (for example at Zenodo);
it is important to use the DOI that points to the specific version, not to the latest version
- Hash (e.g., SHA-256): authors make the artifact available not publicly but only to the committee;
the chairs later have to verify that exactly the reviewed artifact was cited from the article
-
2. Chair sends the artifact notifications.
In your notification message, please include:
- the deadline for final-artifact submission,
- whether an additional page to describe the artifact is allowed in the article, and
- that the artifact should be prominently cited in the article via the DOI.
-
3. Chair provides us (to info@conference-publishing.com)
a mapping from paper id to the according results of the evaluation of the corresponding artifacts.
-
4. ConfPub adds the badges to the articles according to the evaluation results from the chairs above.
-
5. ConfPub adds the "Artifact Available" badge to all articles for which the authors
(1) published their artifact in an archive,
(2) provided the DOI to us (or we assign the DOI in case the artifact wil be published in the ACM DL),
(3) submitted the meta data that are necessary for ACM to connect the article with the artifact to us, and
(4) granted ACM the publishing rights for the artifact (if the artifact is publishing in the ACM DL).
Tasks and Responsibilities (Author <---> ConfPub)
-
Authors complete the submission form with meta-data about their artifact incl. the DOI.
The URL for the submission form can be found on the submission page for the paper
(under heading "Optional Artifact Submission").
If the authors want to publish the artifact in the ACM DL, then the authors inform ConfPub and request a DOI for the
artifact.
-
ConfPub establishes the link from the artifact to the article via the DOI,
adds the "Artifact Available" badge to the paper,
and delivers the artifacts to ACM for publication in the ACM Digital Library.
General Information for Authors
Publication of artifact via ACM DL:
By submitting your artifact to the ACM DL, you ensure
long-term availability, open access, and immutability for your artifact,
and receive a DOI (digital object identifier) for the artifact,
such that your artifact can be cited as publication.
ACM requires only a permission to distribute the artifact in the ACM DL (not a copyright transfer or exclusive license),
which is assigned as part of the publishing-rights agreement for the paper.
Please make sure that you clicked that option (called Auxiliary Material) in your publishing-rights agreement.
If you did not allow ACM to publish your material, please contact us
(and we enable a new publishing-rights form).
Publication of artifact via external provider (e.g., Zenodo):
If you have published your artifact via Zenodo or FigShare already
and you do not want to publish your artifact in the ACM Digital Library,
you still need to complete the metadata fields for the record in the ACM DL.
GitHub or similar repositories are not sufficient to receive an "Artifact Available" badge.
If you host your artifact on GitHub (or similar) so far, you can simply
download the release zip archive from GitHub and upload this file to us (for the ACM DL) or to Zenodo.
(Note that the zip file must contain a license and readme file.)
Or, even simpler, you can publish GitHub releases automatically via Zenodo:
https://guides.github.com/activities/citable-code/
References
Let us know if you have any questions about this process.