The site of the Media Sphera Publishers contains materials intended solely for healthcare professionals.
By closing this message, you confirm that you are a certified medical professional or a student of a medical educational institution.

Login
EN
+7 (495) 482-4118
+7 (495) 482-0604
+8 800 101-59-87
  • (toll-free number for subscriptions)
    Mon-Fri from 10 a.m. to 6 p.m.

  • © 1998-2024
+7 (495) 482-43-29
EN
All journals
Journal
Year
Year
Search result: 0

Nechay T.V.

Pirogov Russian National Research Medical University

Sazhin A.V.

Pirogov Russian National Research Medical University

Loban K.M.

Pirogov Russian National Research Medical University

Bogomolova A.K.

Pirogov Russian National Research Medical University

Suglob V.V.

Pirogov Russian National Research Medical University

Beniia T.R.

Pirogov Russian National Research Medical University

Efficacy and safety of artificial intelligence-based large language models for decision making support in herniology: evaluation by experts and general surgeons

Authors:

Nechay T.V., Sazhin A.V., Loban K.M., Bogomolova A.K., Suglob V.V., Beniia T.R.

More about the authors

Journal: Pirogov Russian Journal of Surgery. 2024;(8): 6‑14

DOI: 10.17116/hirurgia20240816

Views: 635

Downloaded: 59

Download PDF (RU)
Download PDF
Contact the author
Table of contents

To cite this article:

Nechay TV, Sazhin AV, Loban KM, Bogomolova AK, Suglob VV, Beniia TR. Efficacy and safety of artificial intelligence-based large language models for decision making support in herniology: evaluation by experts and general surgeons. Pirogov Russian Journal of Surgery. 2024;(8):6‑14. (In Russ., In Engl.)
https://doi.org/10.17116/hirurgia20240816

Подписка 2025 (Pirogov Russian Journal of Surgery)
МСФ на ЯМ
Close metadata
Recommended articles:
Prospects for the use of robo­tic-assisted technologies in aesthetic plastic surgery. Plastic Surgery and Aesthetic Medi­cine. 2023;(4-2):24-34
Cost of drug therapy for treatment and prevention of complications following chro­nic kidney disease during outpatient hemo­dialysis. Medi­cal Technologies. Asse­ssment and Choice. 2023;(4):52-63
Digi­tal healthcare of the Russian Fede­ration: basic concepts for practitioners. Russian Journal of Preventive Medi­cine. 2024;(1):97-105
Simu­ltaneous hernioplasty (PIRS) in pediatric lapa­roscopic appe­ndectomy. Endo­scopic Surgery. 2024;(1):55-59
Comparison of the predictive values of traditional cardiovascular risk asse­ssment methods using SCORE and FRAMINGHAM scales, machine learning technologies «INTEREPID». Russian Journal of Preventive Medi­cine. 2024;(2):96-102
Feasibility and introduction of asse­ssing the quality of care into info­rmation system of the Piro­gov Moscow City Clinical Hospital No. 1. Medi­cal Technologies. Asse­ssment and Choice. 2024;(1):17-25
Comparison between Lichtenstein hernioplasty and lapa­roscopic transabdominal preperitoneal repair acco­rding to quality of life. Russian Journal of Evidence-Based Gastroenterology. 2024;(1):39-47
First expe­rience of using CAD EYE technology. Arti­ficial inte­lligence-assisted colo­noscopy in children. (A series of clinical cases). Russian Journal of Evidence-Based Gastroenterology. 2024;(1):94-100
Deve­lopment and testing of new methodical approaches for predicting cardiovascular events in healthy people using machine learning technology based on the «INTEREPID» inte­rnational research. Russian Journal of Preventive Medi­cine. 2024;(3):72-79
Rationale of reco­mmendations for the dehy­dratation prevention in mili­tary personnel in hot and dry climates based on the sports medi­cine expe­rience. Russian Journal of Preventive Medi­cine. 2024;(3):86-92

OBJECTIVE

To evaluate the quality of recommendations provided by ChatGPT regarding inguinal hernia repair.

MATERIAL AND METHODS

ChatGPT was asked 5 questions about surgical management of inguinal hernias. The chat-bot was assigned the role of expert in herniology and requested to search only specialized medical databases and provide information about references and evidence. Herniology experts and surgeons (non-experts) rated the quality of recommendations generated by ChatGPT using 4-point scale (from 0 to 3 points). Statistical correlations were explored between participants’ ratings and their stance regarding artificial intelligence.

RESULTS

Experts scored the quality of ChatGPT responses lower than non-experts (2 (1—2) vs. 2 (2—3), p<0.001). The chat-bot failed to provide valid references and actual evidence, as well as falsified half of references. Respondents were optimistic about the future of neural networks for clinical decision-making support. Most of them were against restricting their use in healthcare.

CONCLUSION

We would not recommend non-specialized large language models as a single or primary source of information for clinical decision making or virtual searching assistant.

Keywords:

clinical decision making support tool
large language model
ChatGPT
guidelines
evidence level
inguinal hernia
artificial intelligence

Authors:

Nechay T.V.

Pirogov Russian National Research Medical University

Sazhin A.V.

Pirogov Russian National Research Medical University

Loban K.M.

Pirogov Russian National Research Medical University

Bogomolova A.K.

Pirogov Russian National Research Medical University

Suglob V.V.

Pirogov Russian National Research Medical University

Beniia T.R.

Pirogov Russian National Research Medical University

Received:

10.05.2024

Accepted:

28.05.2024

List of references:

  1. Jia Z, Chen J, Xu X, Kheir J, Hu J, Xiao H, Peng S, Hu XS, Chen D, Shi Y. The importance of resource awareness in artificial intelligence for healthcare. Nature Machine Intelligence. 2023;5(7):687-698.  https://doi.org/10.1038/s42256-023-00670-0
  2. Calaluce R, Davis JW, Bachman SL, Gubin MM, Brown JA, Magee JD, Loy TS, Ramshaw BJ, Atasoy U. Incisional hernia recurrence through genomic profiling: a pilot study. Hernia. 2013;17(2):193-202.  https://doi.org/10.1007/s10029-012-0923-4
  3. Venigalla A, Frankle J, Carbin M. BioMedLM: a Domain-Specific Large Language Model for Biomedical Text. 2022. Accessed June 15, 2023. https://www.mosaicml.com/blog/introducing-pubmed-gpt
  4. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P, Seneviratne M, Gamble P, Kelly C, Babiker A, Schärli N, Chowdhery A, Mansfield P, Demner-Fushman D, Agüera Y Arcas B, Webster D, Corrado GS, Matias Y, Chou K, Gottweis J, Tomasev N, Liu Y, Rajkomar A, Barral J, Semturs C, Karthikesalingam A, Natarajan V. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180.  https://doi.org/10.1038/s41586-023-06291-2
  5. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence. 2023;6:1169595. https://doi.org/10.3389/frai.2023.1169595
  6. Homolak J. Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma. Croatian Medical Journal. 2023;64(1):1-3.  https://doi.org/10.3325/cmj.2023.64.1
  7. Heath R. ChatGPT, in study, achieves a 72% success. Axios. August 29, 2023. Accessed September 12, 2023. https://www.axios.com/2023/08/29/chatgpt-medical-diagnosis-study
  8. Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, Landman A, Dreyer K, Succi MD. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. Journal of Medical Internet Research. 2023;25:e48659. https://doi.org/10.2196/48659
  9. Owusu T. Doctors should think twice about using ChatGPT for medical advice, here’s why. The Street. December 5, 2023. Accessed January 15, 2024. https://www.thestreet.com/technology/doctors-should-think-twice-about-using-chatgpt-for-medical-advice-heres-why
  10. Can doctors trust chatgpt? No, but they use it anyway. ICT&health. Published online August 7, 2023. Accessed August 24, 2023. https://ictandhealth.com/can-doctors-trust-chatgpt-no-but-they-use-it-anyway/news
  11. Taylor J. AMA calls for stronger AI regulations after doctors use ChatGPT to write medical notes. The Guardian. July 27, 2023. Accessed August 20, 2023. https://www.theguardian.com/technology/2023/jul/27/chatgpt-health-industry-hospitals-ai-regulations-ama
  12. Mello MM, Guha N. ChatGPT and Physicians’ Malpractice Risk. JAMA Health Forum. 2023;4(5):e231938. https://doi.org/10.1001/jamahealthforum.2023.1938
  13. Hetrick C. Why doctors using ChatGPT are unknowingly violating HIPAA. USC Price School of Public Policy. July 7, 2023. Accessed August 27, 2023. https://priceschool.usc.edu/news/chatgpt-doctors-data-privacy-hipaa
  14. Omelyanovskiy VV, Gorkavenko FV, Ignatyeva NV, Mukhortova PA, Ryzhova OR, Teptsova TS, Kingshott AA, Kondratyeva BB. Real-world data: principles of use in decision making and in health technology assessment. FARMAKOEKONOMIKA. Modern Pharmacoeconomics and Pharmacoepidemiology. 2023;16(2):283-290. (In Russ.). https://doi.org/10.17749/2070-4909/farmakoekonomika.2023.167
  15. Bittner R, Bingener-Casey J, Dietz U, Fabian M, Ferzli G, Fortelny R, Köckerling F, Kukleta J, LeBlanc K, Lomanto D, Misra M, Morales-Conde S, Ramshaw B, Reinpold W, Rim S, Rohr M, Schrittwieser R, Simon T, Smietanski M, Stechemesser B, Timoney M, Chowbey P. Guidelines for laparoscopic treatment of ventral and incisional abdominal wall hernias (International Endohernia Society (IEHS)—Part 3. Surgical Endoscopy. 2014;28(2):380-404.  https://doi.org/10.1007/s00464-013-3172-4
  16. Simons MP, Aufenacker T, Bay-Nielsen M, Bouillot JL, Campanelli G, Conze J, de Lange D, Fortelny R, Heikkinen T, Kingsnorth A, Kukleta J, Morales-Conde S, Nordin P, Schumpelick V, Smedberg S, Smietanski M, Weber G, Miserez M. European Hernia Society guidelines on the treatment of inguinal hernia in adult patients. Hernia. 2009;13(4):343-403.  https://doi.org/10.1007/s10029-009-0529-7
  17. Bittner R, Bingener-Casey J, Dietz U, Fabian M, Ferzli GS, Fortelny RH, Köckerling F, Kukleta J, Leblanc K, Lomanto D, Misra MC, Bansal VK, Morales-Conde S, Ramshaw B, Reinpold W, Rim S, Rohr M, Schrittwieser R, Simon T, Smietanski M, Stechemesser B, Timoney M, Chowbey P; International Endohernia Society (IEHS). Guidelines for laparoscopic treatment of ventral and incisional abdominal wall hernias (International Endohernia Society (IEHS)—Part 1. Surgical Endoscopy. 2014;28(1):2-29.  https://doi.org/10.1007/s00464-013-3170-6
  18. Fitzgibbons RJ Jr, Giobbie-Hurder A, Gibbs JO, Dunlop DD, Reda DJ, McCarthy M Jr, Neumayer LA, Barkun JS, Hoehn JL, Murphy JT, Sarosi GA Jr, Syme WC, Thompson JS, Wang J, Jonasson O. Watchful Waiting vs Repair of Inguinal Hernia in Minimally Symptomatic Men. JAMA. 2006;295(3):285.  https://doi.org/10.1001/jama.295.3.285
  19. Köckerling F, Bittner R, Kuthe A, Hukauf M, Mayer F, Fortelny R, Schug-Pass C. TEP or TAPP for recurrent inguinal hernia repair—register-based comparison of the outcome. Surgical Endoscopy. 2017;31(10):3872-3882. https://doi.org/10.1007/s00464-017-5416-1
  20. De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, Rizzo C. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Frontiers in Public Health. 2023;11:1166120. https://doi.org/10.3389/fpubh.2023.1166120
  21. Janssen BV, Kazemier G, Besselink MG. The use of ChatGPT and other large language models in surgical science. BJS Open. 2023;7(2):zrad032. https://doi.org/10.1093/bjsopen/zrad032
  22. Samaan JS, Yeo YH, Rajeev N, Hawley L, Abel S, Ng WH, Srinivasan N, Park J, Burch M, Watson R, Liran O, Samakar K. Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery. Obesity Surgery. 2023;33(6):1790-1796. https://doi.org/10.1007/s11695-023-06603-5
  23. Ayoub NF, Lee YJ, Grimm D, Balakrishnan K. Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient Instructions. JAMA Otolaryngology — Head & Neck Surgery. 2023;149(6):556.  https://doi.org/10.1001/jamaoto.2023.0704
  24. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A, Schulman J, Hilton J, Kelton F, Miller L, Simens M, Askell A, Welinder P, Christiano P, Leike J, Lowe R. Training language models to follow instructions with human feedback. Arxiv. March 4, 2022. Accessed August 11, 2023. https://arxiv.org/abs/2203.02155
  25. Haupt CE, Marks M. AI-Generated Medical Advice — GPT and Beyond. JAMA. 2023;329(16):1349. https://doi.org/10.1001/jama.2023.5321
  26. Jones E, Steinhardt J. Capturing Failures of Large Language Models via Human Cognitive Biases. Arxiv. February 24, 2022. Accessed August 18, 2023. https://arxiv.org/abs/2202.12299
  27. Kirchner JH. Cognitive Biases in Large Language Models. On Brains, Minds, And Their Possible Uses. September 25, 2021. Accessed August 12, 2023. https://universalprior.substack.com/p/cognitive-biases-in-large-language?utm_campaign=post&utm_medium=web

Close metadata

Contact the author
Email
Message
The publishing house "Media Sphere" does not provide treatment services, does not give recommendations on contacting medical institutions and specific specialists, on the prescription of medicines and dietary supplements.

Email Confirmation

An email was sent to test@gmail.com with a confirmation link. Follow the link from the letter to complete the registration on the site.

Email Confirmation

Contact us
If you have any questions, complaints or suggestions, please use this feedback form.
Email
Message
The publishing house "Media Sphere" does not provide treatment services, does not give recommendations on contacting medical institutions and specific specialists, on the prescription of medicines and dietary supplements.
Submit Application

You can become an author of one of our magazines, send a request and we will consider it in during the day.

Name
Phone
E-mail
Message
Login
Registration
E-mail
Password
Forgot Password?

Log in to the site using your account in one of the services

Password recovery
E-mail
Login
Register


We use cооkies to improve the performance of the site. By staying on our site, you agree to the terms of use of cооkies. To view our Privacy and Cookie Policy, please. click here.