1. Open Research Objects (ORO)

Indicates that author-created digital objects (data/code) are permanently stored in a public repository with a globally unique identifier, guaranteed persistence, and an open license to maximize access. Aligns with ACM “Artifacts Available” and COS “Open Data/Materials” for digital objects.

The AEC and authors decide what objects are “relevant”. Provide enough documentation for reviewers to understand core functionality and data context.

Review criteria

• Availability: Public repository with global identifier and guaranteed persistence.
• License: Open license maximizing accessibility (e.g., OSI/CC/public domain).
• Authorship: Reasonable and complete authorship attached to the archived artifact.
• Documentation: For software: core functionality overview. For data: context, description, source, and usage in the paper. Metadata accessible for relevant physical objects.

2. Reusable/Research Objects Reviewed (ROR)

Higher-level than ORO; requires ORO. Corresponds to IEEE “Code Reviewed.” All author-created digital objects used in the research (data/software) are reviewed.

Software review criteria

• Documentation: Statement of need/function, install instructions, usage examples, API docs.
• Functionality: Reviewers can install and verify core functionality on commodity hardware with step-by-step instructions.
• Testing: Documented manual checks (e.g., sample input & expected output) and ideally an automated test suite with CI (e.g., GitHub Actions).

Data review criteria

• Documentation: Dataset context, description, source, and how it’s used in the paper.
• Functionality: Enable reuse beyond the paper; for proprietary formats, include code to access data programmatically.

3. Results Reproduced (ROR-R)

Highest tier, awarded when evaluators reproduce the paper’s key results using the authors’ objects, methods, code, and analysis conditions. Focus is on reproducing behavior/claims, not exact values (especially with hardware dependence).

Scope

• Reproduce behavior: Attempt on the same hardware where feasible; otherwise collaborate to establish equivalent behavior on commodity hardware.
• Reproduce central results: AEC identifies key results/claims; badge is based on reproducing these behaviors.

Additional requirements

• Paper already qualified for ROR (and thus ORO).
• Authors assist with materials and clarifications.

Process

1. Authors submit a comprehensive reproduction plan and research objects.
2. Independent attempt to reproduce by evaluators.
3. Author–evaluator collaboration to resolve discrepancies.
4. Final verification by the AEC; ROR-R badge appears next to ROR in proceedings.