Skip to main content

Reviewing Research Rigor

Use when a structurally validated research artifact needs independent semantic review before publication, decision-making, or final delivery.

Independence Contract​

Begin in a fresh context. This is an artifact-only review: use only the supplied ARA, its Level 1 receipt, and any explicitly bounded source packet. Use no external browsing, retrieval, remembered drafting rationale, or author conversation as evidence. If the reviewer helped create the artifact, disclose the conflict and require another fresh reviewer for the final gate.

Level 1 must already be valid. If it is missing or invalid, stop and report the prerequisite failure rather than performing semantic review.

Read Order​

  1. PAPER.md
  2. logic/claims.md
  3. logic/experiments.md
  4. logic/problem.md
  5. logic/concepts.md
  6. solution architecture, algorithm, constraints, and heuristics
  7. logic/related_work.md
  8. trace/exploration_tree.yaml
  9. evidence/README.md
  10. all evidence needed to test material claims, with at least three spot checks when more exist

Record the actual order in the report.

Review Dimensions​

Score each dimension from 1–5:

  • D1 Evidence relevance: Does each proof substantively test its claim, using type-appropriate evidence?
  • D2 Falsifiability: Are failure criteria specific, actionable, scope-matched, and independently testable?
  • D3 Scope calibration: Do claims stay within the evidence, assumptions, and declared boundaries?
  • D4 Argument coherence: Do observations lead to gaps, insight, solution, claims, and evidence without contradiction?
  • D5 Exploration integrity: Are decisions, alternatives, pivots, and failures specific and honestly supported rather than reconstructed as fact?
  • D6 Methodological rigor: Are baselines, ablations, statistical reporting, metrics, and reproduction detail adequate for the claim type?

Use calibrated scoring: 5 is excellent, 3 is adequate with material limitations, and 1 is fundamentally unsound.

Findings Contract​

Every finding includes:

  • sequential ID;
  • dimension;
  • severity: critical, major, minor, or suggestion;
  • target file and entity;
  • exact quote in evidence_span when the issue concerns present content;
  • factual observation;
  • reasoning;
  • actionable fix.

For an absence, omit the evidence span and name what is missing. If no change is required, explicitly state “none needed” for that dimension. Sort findings by severity.

Grades​

  • Strong Accept: mean ≥ 4.5 and no dimension < 3
  • Accept: mean ≥ 3.8 and no dimension < 2
  • Weak Accept: mean ≥ 3.0 and no dimension < 2
  • Weak Reject: mean ≥ 2.0 and either mean < 3.0 or any dimension < 2
  • Reject: mean < 2.0 or any dimension = 1

Output​

Write <artifact-dir>/level2_report.json. The report must repeat the artifact_digest from the validated Level 1 receipt so the coordinator can bind the verdict to the reviewed bytes:

{
  "artifact": "name",
  "artifact_dir": "/absolute/path",
  "artifact_digest": "sha256 digest from level1_receipt.json",
  "review_version": "1.0.0",
  "review_independence": {
    "fresh_context": true,
    "artifact_only": true,
    "external_sources_used": false,
    "conflict_disclosed": false
  },
  "prerequisite": {
    "level1_valid": true,
    "receipt_path": "/absolute/path/level1_receipt.json"
  },
  "overall": {
    "grade": "Accept",
    "mean_score": 4.0,
    "one_line_summary": "...",
    "strengths_summary": [],
    "weaknesses_summary": []
  },
  "dimensions": {},
  "findings": [],
  "questions_for_authors": [],
  "read_order": []
}

Write syntactically valid JSON and ensure every quoted evidence_span is copied exactly from the target file. Return the report path to the coordinator. The coordinator—not this restricted reviewer profile—runs the deterministic report validator and records level2_validation_receipt.json.

Restrictions​

  • Do not modify the artifact under review.
  • Do not execute its code.
  • Do not fetch cited sources unless the coordinator starts a separately scoped source-verification mission.
  • Do not substitute agreement elsewhere in the artifact for evidence.
  • Do not soften a finding because the deadline is near.
  • Do not claim independent review if context or memory from authorship was available.

Source: jknash/hermes-shared-skills · branch hermes-jkdev001 @ 1d0d545c3970 · skills/research/reviewing-research-rigor/ · view source · Imported 2026-10-04. Supporting files (references, scripts) remain in the source repository.

version 1.0.0 · license MIT.

Published by Muse · 2026-10-04.