How to Remove Duplicate References

Combine records from all databases in one reference manager and run its duplicate-detection tool. Then check borderline matches manually, merge or remove true duplicates, and record how many were removed so the PRISMA flow diagram can report accurate counts.

Steps

1. Export each database's results in a standard format and import them into one library (keep a copy of the raw exports).
2. Run automatic de-duplication in your reference manager. Tools match on fields such as title, authors, year and DOI; settings differ between programs.
3. Review the proposed duplicate sets. Be careful with similar titles from different papers, conference abstracts and their full papers, and records with missing fields.
4. Merge or remove true duplicates and keep the more complete record.
5. Run a second pass with sorting by title or DOI to catch records the tool missed.
6. Record the numbers: records identified, duplicates removed and records screened.

Published guidance describes a stepwise manual approach for EndNote that is also a good model for other tools.

Why it matters

Duplicates inflate the number of records to screen and, if not caught, can cause the same study to be counted twice in a review. Different reports of the same study (for example a trial and its follow-up paper) are different issue: link them as one study during data extraction.

Illustrative example

Invented for teaching. A team searches four databases and gets 4,300 records. After automatic de-duplication, 3,100 remain. A manual pass removes 85 more, leaving 3,015 records to screen. They record each number for the flow diagram.

Common pitfalls

• Deleting records instead of marking them, which makes counts hard to audit.
• Treating different papers about the same study as duplicates and losing data.

Key takeaways

• De-duplicate, check manually and record counts.
• Keep the raw exports.

This page is general educational information about research methods. It is not medical, legal or regulatory advice; follow your institution's ethics board and your target journal's instructions.