AI for Literature Review Data Extraction: Tools, Benefits, and Limitations

Generative AI, exemplified by tools like ChatGPT, holds considerable potential for automating data extraction in literature reviews. These advanced language models can process large volumes of text and identify specific pieces of information. Research indicates that generative AI tools are adept at extracting structured data such as publication year, country of origin, and numerical figures like participant counts. The NICE Position Statement on AI in evidence generation acknowledges that LLMs can be employed to automate data extraction from both quantitative and qualitative studies by inputting specific prompts to generate desired outputs. This capability promises to significantly reduce the manual effort involved in collecting data for systematic literature reviews.

Beyond generative AI, a range of AI tools are available to assist with data extraction, often by providing quick summaries of PDF articles. Tools like Google Scholar PDF Reader can offer a rapid overview of an article's key points, potentially saving researchers considerable time. Many free AI tools offer PDF summarization, but users should be aware of limitations such as daily usage caps on the number of documents analyzed or questions asked. Some tools, like PdfGear, offer unlimited analysis, while others, like Google Scholar PDF Reader, are free extensions that integrate with browsers to highlight key information, though they may lack advanced text highlighting features.

Elicit functions by extracting data from uploaded PDFs and presenting it in an organized table. Users can extract up to 24 different types of information, including findings, participant details, location, and outcomes measured. This tabular presentation significantly enhances the synthesis of articles. Elicit's effectiveness is often linked to specific prompt structures, such as 'What are the effects of ___ on ___?'. While Elicit can save researchers substantial time, its free basic version allows data extraction and paper uploads, with priced versions offering paper summaries and export options in CSV and bib formats.

A recurring theme in the evaluation of AI for data extraction is the critical need for human oversight. Research consistently concludes that AI tools, while powerful aids, are not yet reliable enough to be trusted without human involvement. Human reviewers are essential for validating AI-generated content, ensuring accuracy, contextual relevance, and compliance with research standards. Even tools like Elicit, which strive for accuracy, have been found to contain approximately 10% incorrect information, necessitating close checking by users. The process of sifting through irrelevant information also remains a task that requires human judgment and effort.