Library Seminars on Reference Management Tools
Visit the Library Video Hub to find recorded webinars of seminars about EndNote, Mendeley and Zotero.
Open Access Evidence Synthesis Resources
Free evidence synthesis resources for researchers and librarians collected by Cornell University Library
Manage & Analyse your Literature
A citation management program will save you a lot of time when doing your evidence synthesis. Programs like Endnote, Mendeley or Zotero will store and organize the citations collected during your screening, de-duplicate the results and automatically format in-text citations and bibliographies in your manuscript.
With Mendeley Reference Manager you can store, organize and search all your references from just one library, seamlessly insert references and bibliographies into documents, cite, read, highlight and annotate PDFs. Download Mendeley.
Zotero is a free and open-source reference management software to manage bibliographic data and related research materials. Download Zotero.
EndNote enables you to easily view, edit, organize, and share research material. You can Insert in-text citations while simultaneously creating a bibliography, easily collaborate, read, review, annotate and search PDFs in your library. You can download the corporate licensed version of ENDNOTE here.
The purpose of article screening is to remove studies that are clearly not related to your topic. This critical appraisal will help you evaluate the quality of relevant studies, as not all will have a sufficiently rigorous methodology to avoid biased results. Use your inclusion/exclusion criteria to first screen the title and abstracts of your studies and determine whether they are relevant to your research question. Once titles and abstracts have been screened, the full text must be retrieved and screened to definitely decide whether the study fits the eligibility criteria of your synthesis.
It is highly recommended that two independent reviewers screen all studies, resolving areas of disagreement by consensus or by a third party who is an expert in the field. Listed below are tools that can be used for article screening.
Excel
Lists of references can be exported from reference management tools such as EndNote into Excel for screening. This is the most basic tool for the job. For a more fully-featured product see Rayyan below.
Rayyan
Rayyan is a free online tool that can be used for screening and coding of studies. It will pre-populate inclusion and exclusion criteria, although these can be customised. You can also tag and filter to code and organize references. Works well for title and abstract screening, for full-text screening revert to EndNote (you can export references from Rayyan back to EndNote for this step). Supports collaboration and blind reviewing to avoid bias. Has an app for use on mobiles and tablets.
-
A comprehensive guide produced by McGill University on using Rayyan for systematic reviews. Please note that the authors can only offer advice to staff and students at McGill.
Covidence
This is an online systematic review management tool. Koç University does not have a licence for it, but individual licences can be purchased for small teams.
-
A comprehensive guide from University of North Carolina on using Covidence for systematic reviews.
SRDR
SRDR (Systematic Review Data Repository) is a Web-based tool for the extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data. Access the "Create an Extraction Form" section for more information.
The Systematic Review Toolbox
The SR Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. Use the advanced search option to restrict to tools specific to data extraction.
“My sources are unreliable, but their information is fascinating” Ashleigh Brilliant
Unreliable sources of fascinating information can be great fun but, of course, high quality information is vital to sound research. Before deciding whether or not to incorporate what you have found into your literature review, you need to evaluate the resources to make sure that they contain information which is valuable and pertinent.
Accuracy
- Is the information reliable?
- Is the information error-free?
- Is the information based on proven facts?
- Can the information be verified against other reliable sources?
Authority
- Who is the author?
- Does he or she have the qualifications to speak/write on that topic?
- Is the author affiliated with a reputable university or organization in this subject field?
- Is the source peer reviewed or refereed?
Objectivity
- What is the intended purpose of the information?
- Is the information fact or opinion?
- Is the information biased?
Currency
- When was the information published?
- Is the information current or out-dated?
- Does currency matter in this topic?
Coverage
- Does the information covered meet your information needs?
- Does it provide basic or in depth coverage
See the following links for further help on critically appraising the literature:
- How to critically appraise articles
This video from Idaho State University Library describes the steps and tools for critically appraising articles to use in a review.
- Critical appraisal of quantitative research
Chapter 4 from the Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions.
- Cochrane Handbook for Systematic Reviews of Interventions
This handbook includes chapters on assessing bias in included studies.
- Critical appraisal of clinical trials
An interactive presentation covering the most important features of well controlled trials.
- Critical appraisal notes and checklists
Produced by the Scottish Intercollegiate Guidelines Network (SIGN) these checklists give a framework of questions to apply to each study to judge the likelihood of bias.
- Critical Appraisal Skills Programme (CASP) checklists
Critical appraisal tools to help you evaluate research.
Data Extraction
You will need to extract data from relevant studies in order to examine and compare results. While the data is being extracted, it is very important to employ good data management practices. Proper data management should begin as soon as you start extracting data, and may even dictate which types of data you decide to retain. Some reviews require the use of software to help with extracting data.
What should I extract?
You should plan to extract data that is relevant to answering the question posed in your systematic review. As mentioned previously, it may help to consult other similar systematic reviews to identify what data to collect. You should use your key questions and your eligibility criteria as a starting point.
Helpful data may include:
- Information about the study (author(s), year of publication, title, DOI)
- Demographics (age, sex, ethnicity, diseases / conditions, other characteristics related to the intervention / outcome)
- Methodology (study type, participant recruitment / selection / allocation, level of evidence, study quality)
- Intervention (quantity, dosage, route of administration, format, duration, time frame, setting)
- Outcomes (quantitative and / or qualitative)
If you plan to synthesize data, you will want to collect additional information such as sample sizes, effect sizes, dependent variables, reliability measures, pre-test data, post-test data, follow-up data, and statistical tests used.
Data Extraction Tools
You can use spreadsheet or database software to create custom extraction forms. Spreadsheet functions such as drop-down menus and range checks can speed up the process and help prevent data entry errors. Relational databases (such as Microsoft Access) can help you extract information from different categories like citation details, demographics, participant selection, intervention, outcomes, etc.
RevMan is free software used to manage Cochrane reviews. For more information on RevMan, including an explanation of how it may be used to extract and analyze data, watch Introduction to RevMan - a guided tour.
Survey or form tools can help you create custom forms with many different question types, such as multiple choice, drop downs, ranking, and more. Content from these tools can often be exported to spreadsheet or database software as well. Here at KU we have access to the survey/form software Qualtrics.
SRDR (Systematic Review Data Repository) is a Web-based tool for the extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data. Access the "Create an Extraction Form" section for more information.
The SR Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. Use the advanced search option to restrict to tools specific to data extraction.
DistillerSR is a systematic review management software program, similar to Covidence. It guides reviewers in creating project-specific forms, extracting, and analyzing data.