A comparison of a literature review by ChatGPT and expert writers

ChatGPT- an innovative content generation tool

ChatGPT (Generative Pre-trained Transformer), developed by OpenAI, is an artificial intelligence language tool that used machine learning algorithms to develop text (Ariyaratne, 2023). Those who adopted ChatGPT early have praised the tool for the content it develops. This is due to the fact that Large Language Models (LLMs) are used which are trained on extensive data to predict the next best text that generates content that reads naturally. ChatGPT produces content that is coherent and difficult to distinguish from the text written by humans. University students are increasingly using ChatGPT for writing their assignments, literature reviews and dissertations.

However, there are numerous articles that highlight the end of traditional assignments, as ChatGPT could generate high-scoring papers and stimulate critical thinking. The ethical and permissible limits of ChatGPT’s use in scientific writing are still unknown (Gao, 2022).

Studies on the reliability of ChatGPT content

Research has been conducted on determining the capabilities of chatbots with scientific rigour.

• According to Yeadon et al. (2023), ten essays on physics were assessed for academic content, comprehension of the underlying topic and writing style. The results revealed that the essays written by AI were of a high quality, but the quality varies in a way similar to essays written by humans.
• In a study by Herbold et al. (2023) that compared the essays written by Humans and ChatGPT, it was discovered that using ChatGPT resulted in high-quality essays and was effective in writing argumentative essays. The content generated by AI was highly structured and had identical beginnings of the concluding sections, such as “Overall”, “In summary ”and “In conclusion”. On the other hand, human-written essays are looser in structure and represent the guidelines.
• However, researchers have noted the limitation of CharGPT for research purposes: ChatGPT does not produce original ideas; it generates content based on the training data. This phenomenon significantly increases the chances of plagiarism.
• A study conducted by Ariyaratne et al. in 2023 revealed that four out of five papers generated by ChatGPT were erroneous, and one had false references. This raises the question of the authenticity of using ChatGPT for academic research. Kleebayon (2023) noted that ChatGPT has a tendency to give a biased content.
• Moreover, scholars argue that when CharGPT cannot co-author a research, it is unethical to use ChatGPT for literature review and other academic research (Haman, 2023).

ChatGPT vs Expert Writers

Liao and Sundar (2023) conducted an experiment that describes the MAIN model for communication. The following is an excerpt from the study, serving as proof that ChatGPT cannot be used for conducting a literature review:

“Prompt: Describe the MAIN Model by Shyam Sundar

ChatGPT: The MAIN Model, developed by S. Shyam Sundar, is a theoretical framework that explains how people process and make sense of information they encounter online and how the design of digital media platforms shapes this process. The acronym MAIN stands for Media Affordances and Heuristics-based Information Processing. The model posits that the design of digital media platforms, specifically the technological affordances of the platform, shapes the cognitive heuristics that users employ when processing information. Technological affordances are the capabilities and possibilities that a platform provides for users to interact with information, such as browsing, searching, and filtering. Cognitive heuristics are mental shortcuts that people use to simplify decision-making and information processing, such as the availability heuristic, the representativeness heuristic, and the affect heuristic. The MAIN model suggests that the design of digital media platforms affects the way people process information, as different affordances and heuristics can lead to different information processing outcomes. For example, a platform with strong search affordances may lead users to employ the availability heuristic and rely on the first results they find, while a platform with strong social affordances may lead users to employ the affect heuristic and rely on the opinions of others.”

For the uninitiated, the answer seems correct. However, those who have specialised in the field can discern its accuracy. Likewise, if a student refers exclusively to ChatGPT without referring to academic publications for their literature review, they may get the concepts wrong and their literature review may be adversely affected.

Here is the content for the MAIN model, written by a human:

MAIN model, conceptualised by Shyam Sundar, is an acronym for the four key purposes of media that can have psychological effects on the audience- Modality (M), Agency (A), Interactivity (I) and Navigability (N). Although the source and content of digital media are crucial in establishing ascribed credibility, the MAIN model focuses on the technological characteristics of digital media that can influence credibility judgements.

Modality is related to the structure of the media and is the most apparent. The concept of modality is closely related to the concept of medium because, traditionally, media vary according to modality, with print being largely textual, radio being aural, and television being audiovisual. An agency, or agent, is the source, at least on a psychological level, especially when there is no other credited source for a specific piece of information. Interactivity denotes both interaction and activity. Interactivity also implies that the medium is responsive to user demands and capable of accounting for variations in user input during the course of the engagement. By presenting the user with steering wheels, navigability tools on the interface can instil a sense of self as a source. Modern media provides users with numerous options for determining how to enter and navigate a mediated space (Sundar, 2015).

Conclusion

ChatGPT is a type of Artificial Intelligence tool based on Large Language Models that have been trained to generate human-like content. It has gained significant interest from many users for its content. Research has shown that ChatGPT can produce structured essays. However, there were some concerns regarding the sources, plagiarism and the quality of content. From the comparison between ChatGPT-generated content and human content, it is evident that the content from ChatGPT is incorrect. Therefore, students must exercise caution before blindly using ChatGPT for assistance with their assignments and dissertations.  Nevertheless, they can use ChatGPT for help with structuring their content.

About Tutors India

At Tutors India, we are a team of academic researchers and writers who assist students with writing a literature review. Our team is diverse, enabling us to guide students specialising in various disciplines. We ensure selecting articles from reputed journals and abide by the university guidelines for assignments, essays and dissertations. Moreover, we proofread and edit the content, ensuring the review is error- and plagiarism-free.

To know more about how a literature review is written in various fields, check out our literature review examples.

References

1. Ariyaratne, S., Iyengar, K.P., Nischal, N. et al., 2023.A comparison of ChatGPT-generated articles with human-written articles. Skeletal Radiol 52, 1755–1758.
2. Gao, C.A., Howard, F.M., Markov, N.S., Dyer, E.C., Ramesh, S., Luo, Y. and Pearson, A.T., 2022. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. BioRxiv, pp.2022-12.
3. Yeadon, W., Inyang, O.O., Mizouri, A., Peach, A. and Testrow, C.P., 2023. The death of the short-form physics essay in the coming AI revolution. Physics Education, 58(3), p.035027.
4. Herbold, S., Hautli-Janisz, A., Heuer, U. et al. A large-scale comparison of human-written versus ChatGPT-generated essays. Sci Rep13, 18617.
5. Kleebayoon, A. and Wiwanitkit, V., 2023. ChatGPT-generated articles and human-written articles: correspondence. Skeletal Radiology, 52(12), pp.2493-2493.
6. Michael Haman& Milan Školník  Using ChatGPT to conduct a literature review, Accountability in Research
7. Sundar, S.S. and Liao, M., 2023. Calling BS on ChatGPT: Reflections on AI as a Communication Source. Journalism & Communication Monographs, 25(2), pp.165-180.
8. Sundar, S.S., Jia, H., Waddell, T.F. and Huang, Y., 2015. Toward a theory of interactive media effects (TIME) four models for explaining how interface features affect user psychology. The handbook of the psychology of communication technology, pp.47-86.