How to Resolve Disagreements During Study Screening

Disagreements between reviewers are normal. Resolve them through structured discussion, arbitration by a third reviewer if needed, and clarification of criteria so the same issue does not recur. Record how conflicts were handled and, if you wish, report agreement.

A step-by-step approach

1. List conflicts after both reviewers finish independently.
2. Check for simple errors such as a slip or an overlooked section of the paper.
3. Discuss in terms of the criteria, not opinions: which criterion decides this?
4. Ask a third reviewer to decide if the two cannot agree.
5. Update the decision guide with the example so future cases are consistent.
6. Record the final decision and who made it.

Measuring agreement

Many reviews report percent agreement or a chance-corrected statistic such as Cohen's kappa for the screening stage. These numbers show reliability of the process; they do not replace resolving the conflicts. Decide in your protocol whether and how you will report them.

When disagreement is common

A high rate of disagreement suggests unclear criteria. Pause, revise the wording with examples, and re-screen a sample.

Illustrative example

Invented for teaching. Two reviewers disagree on 25 of 400 full texts, mostly about whether "outpatient clinic" counts as primary care. They agree to a definition, update the form, re-check the affected records and record the clarification in the review's methods.

Examples of useful rules

Agree in the protocol that a third reviewer will arbitrate, and who that will be. Decide whether conflicts will be discussed in pairs or in a group. When the disagreement reveals a gap in the criteria, resolve the individual record and then amend the guide, checking earlier decisions for the same issue. Report the process in the methods in a sentence or two.

Common pitfalls

• Letting the more senior reviewer decide without discussion.
• Changing eligibility criteria silently.

Key takeaways

• Resolve through criteria, not hierarchy.
• Document how conflicts were settled.

This page is general educational information about research methods. It is not medical, legal or regulatory advice; follow your institution's ethics board and your target journal's instructions.