The site of the Media Sphera Publishers contains materials intended solely for healthcare professionals.
By closing this message, you confirm that you are a certified medical professional or a student of a medical educational institution.

Login
EN
+7 (495) 482-4118
+7 (495) 482-4329
+8 800 250-18-12
  • (toll-free number for subscriptions)
    Mon-Fri from 10 a.m. to 6 p.m.

  • © 1998-2025
+7 (495) 482-43-29
EN
All journals
Journal
Year
Year
Search result: 0

Nechay T.V.

Pirogov Russian National Research Medical University

Sazhin A.V.

Pirogov Russian National Research Medical University

Loban K.M.

Pirogov Russian National Research Medical University

Bogomolova A.K.

Pirogov Russian National Research Medical University

Suglob V.V.

Pirogov Russian National Research Medical University

Beniia T.R.

Pirogov Russian National Research Medical University

Efficacy and safety of artificial intelligence-based large language models for decision making support in herniology: evaluation by experts and general surgeons

Authors:

Nechay T.V., Sazhin A.V., Loban K.M., Bogomolova A.K., Suglob V.V., Beniia T.R.

More about the authors

Journal: Pirogov Russian Journal of Surgery. 2024;(8): 6‑14

DOI: 10.17116/hirurgia20240816

Read: 1947 times

Download PDF (RU)
Download PDF
Contact the author
Table of contents

EFFICACY AND SAFETY OF ARTIFICIAL INTELLIGENCE-BASED LARGE LANGUAGE MODELS FOR DECISION MAKING SUPPORT IN HERNIOLOGY: EVALUATION BY EXPERTS AND GENERAL SURGEONS
EFFICACY AND SAFETY OF ARTIFICIAL INTELLIGENCE-BASED LARGE LANGUAGE MODELS FOR DECISION MAKING SUPPORT IN HERNIOLOGY: EVALUATION BY EXPERTS AND GENERAL SURGEONS

To cite this article:

Nechay TV, Sazhin AV, Loban KM, Bogomolova AK, Suglob VV, Beniia TR. Efficacy and safety of artificial intelligence-based large language models for decision making support in herniology: evaluation by experts and general surgeons. Pirogov Russian Journal of Surgery. 2024;(8):6‑14. (In Russ., In Engl.)
https://doi.org/10.17116/hirurgia20240816

Подписка 2026 Хирургия. Журнал им. Н.И. Пирогова (Pirogov Russian Journal of Surgery)
МСФ на ЯМ
Использование инфографики авторами научных работ
Close metadata
Recommended articles:
Arti­ficial inte­lligence in ultrasound diagnosis of thyroid nodu­les. Piro­gov Russian Journal of Surgery. 2024;(12-2):109-116
Long-term outcomes of lapa­roscopic inguinal hernia repair. Piro­gov Russian Journal of Surgery. 2024;(12):67-73
Perioperative mana­gement of patients with valvular heart disease. Clinical guidelines of the All-Russian Public Orga­nization “Federation of Anesthesiologists and Reanimatologists” (2nd revi­sion, part 1). Russian Journal of Anesthesiology and Reanimatology. 2024;(6):6-22
Is arti­ficial inte­lligence nece­ssary for healthcare system?. Medi­cal Technologies. Asse­ssment and Choice. 2024;(4):40-48
Iatrogenic exte­rnal iliac artery injury following inguinal hernia repair. Russian Journal of Cardiology and Cardiovascular Surgery. 2024;(6):695-698
Comparison of long-term outcomes of different methods of lapa­roscopic inguinal hernia repair (TAPP and TEP). Endo­scopic Surgery. 2024;(6):5-12
Challenges, problems and approaches to healthcare digi­tal technologies enha­ncement. Russian Journal of Preventive Medi­cine. 2024;(12):31-36
A review of modern digi­tal personal identification methods using arti­ficial inte­lligence technology in fore­nsic dentistry. Stomatology. 2024;(6):79-82
Personalized approach to treatment choice using a digi­tal profile of patient with cardiovascular diseases: the features of clinical deci­sion-making support service. Russian Cardiology Bulletin. 2024;(4-2):105-112
Outcomes of using both self-gripping mesh and poly­propylene meshes fixed by tacks in Liechtenstein repair. Russian Journal of Operative Surgery and Clinical Anatomy. 2024;(4-2):53-57
Abstract:

OBJECTIVE

To evaluate the quality of recommendations provided by ChatGPT regarding inguinal hernia repair.

MATERIAL AND METHODS

ChatGPT was asked 5 questions about surgical management of inguinal hernias. The chat-bot was assigned the role of expert in herniology and requested to search only specialized medical databases and provide information about references and evidence. Herniology experts and surgeons (non-experts) rated the quality of recommendations generated by ChatGPT using 4-point scale (from 0 to 3 points). Statistical correlations were explored between participants’ ratings and their stance regarding artificial intelligence.

RESULTS

Experts scored the quality of ChatGPT responses lower than non-experts (2 (1—2) vs. 2 (2—3), p<0.001). The chat-bot failed to provide valid references and actual evidence, as well as falsified half of references. Respondents were optimistic about the future of neural networks for clinical decision-making support. Most of them were against restricting their use in healthcare.

CONCLUSION

We would not recommend non-specialized large language models as a single or primary source of information for clinical decision making or virtual searching assistant.

Keywords:

clinical decision making support tool
large language model
ChatGPT
guidelines
evidence level
inguinal hernia
artificial intelligence

Authors:

Nechay T.V.

Pirogov Russian National Research Medical University

Sazhin A.V.

Pirogov Russian National Research Medical University

Loban K.M.

Pirogov Russian National Research Medical University

Bogomolova A.K.

Pirogov Russian National Research Medical University

Suglob V.V.

Pirogov Russian National Research Medical University

Beniia T.R.

Pirogov Russian National Research Medical University

Received:

10.05.2024

Accepted:

28.05.2024

Close metadata

References:

  1. Jia Z, Chen J, Xu X, Kheir J, Hu J, Xiao H, Peng S, Hu XS, Chen D, Shi Y. The importance of resource awareness in artificial intelligence for healthcare. Nature Machine Intelligence. 2023;5(7):687-698.  https://doi.org/10.1038/s42256-023-00670-0
  2. Calaluce R, Davis JW, Bachman SL, Gubin MM, Brown JA, Magee JD, Loy TS, Ramshaw BJ, Atasoy U. Incisional hernia recurrence through genomic profiling: a pilot study. Hernia. 2013;17(2):193-202.  https://doi.org/10.1007/s10029-012-0923-4
  3. Venigalla A, Frankle J, Carbin M. BioMedLM: a Domain-Specific Large Language Model for Biomedical Text. 2022. Accessed June 15, 2023. https://www.mosaicml.com/blog/introducing-pubmed-gpt
  4. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P, Seneviratne M, Gamble P, Kelly C, Babiker A, Schärli N, Chowdhery A, Mansfield P, Demner-Fushman D, Agüera Y Arcas B, Webster D, Corrado GS, Matias Y, Chou K, Gottweis J, Tomasev N, Liu Y, Rajkomar A, Barral J, Semturs C, Karthikesalingam A, Natarajan V. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180.  https://doi.org/10.1038/s41586-023-06291-2
  5. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence. 2023;6:1169595. https://doi.org/10.3389/frai.2023.1169595
  6. Homolak J. Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma. Croatian Medical Journal. 2023;64(1):1-3.  https://doi.org/10.3325/cmj.2023.64.1
  7. Heath R. ChatGPT, in study, achieves a 72% success. Axios. August 29, 2023. Accessed September 12, 2023. https://www.axios.com/2023/08/29/chatgpt-medical-diagnosis-study
  8. Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, Landman A, Dreyer K, Succi MD. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. Journal of Medical Internet Research. 2023;25:e48659. https://doi.org/10.2196/48659
  9. Owusu T. Doctors should think twice about using ChatGPT for medical advice, here’s why. The Street. December 5, 2023. Accessed January 15, 2024. https://www.thestreet.com/technology/doctors-should-think-twice-about-using-chatgpt-for-medical-advice-heres-why
  10. Can doctors trust chatgpt? No, but they use it anyway. ICT&health. Published online August 7, 2023. Accessed August 24, 2023. https://ictandhealth.com/can-doctors-trust-chatgpt-no-but-they-use-it-anyway/news
  11. Taylor J. AMA calls for stronger AI regulations after doctors use ChatGPT to write medical notes. The Guardian. July 27, 2023. Accessed August 20, 2023. https://www.theguardian.com/technology/2023/jul/27/chatgpt-health-industry-hospitals-ai-regulations-ama
  12. Mello MM, Guha N. ChatGPT and Physicians’ Malpractice Risk. JAMA Health Forum. 2023;4(5):e231938. https://doi.org/10.1001/jamahealthforum.2023.1938
  13. Hetrick C. Why doctors using ChatGPT are unknowingly violating HIPAA. USC Price School of Public Policy. July 7, 2023. Accessed August 27, 2023. https://priceschool.usc.edu/news/chatgpt-doctors-data-privacy-hipaa
  14. Omelyanovskiy VV, Gorkavenko FV, Ignatyeva NV, Mukhortova PA, Ryzhova OR, Teptsova TS, Kingshott AA, Kondratyeva BB. Real-world data: principles of use in decision making and in health technology assessment. FARMAKOEKONOMIKA. Modern Pharmacoeconomics and Pharmacoepidemiology. 2023;16(2):283-290. (In Russ.). https://doi.org/10.17749/2070-4909/farmakoekonomika.2023.167
  15. Bittner R, Bingener-Casey J, Dietz U, Fabian M, Ferzli G, Fortelny R, Köckerling F, Kukleta J, LeBlanc K, Lomanto D, Misra M, Morales-Conde S, Ramshaw B, Reinpold W, Rim S, Rohr M, Schrittwieser R, Simon T, Smietanski M, Stechemesser B, Timoney M, Chowbey P. Guidelines for laparoscopic treatment of ventral and incisional abdominal wall hernias (International Endohernia Society (IEHS)—Part 3. Surgical Endoscopy. 2014;28(2):380-404.  https://doi.org/10.1007/s00464-013-3172-4
  16. Simons MP, Aufenacker T, Bay-Nielsen M, Bouillot JL, Campanelli G, Conze J, de Lange D, Fortelny R, Heikkinen T, Kingsnorth A, Kukleta J, Morales-Conde S, Nordin P, Schumpelick V, Smedberg S, Smietanski M, Weber G, Miserez M. European Hernia Society guidelines on the treatment of inguinal hernia in adult patients. Hernia. 2009;13(4):343-403.  https://doi.org/10.1007/s10029-009-0529-7
  17. Bittner R, Bingener-Casey J, Dietz U, Fabian M, Ferzli GS, Fortelny RH, Köckerling F, Kukleta J, Leblanc K, Lomanto D, Misra MC, Bansal VK, Morales-Conde S, Ramshaw B, Reinpold W, Rim S, Rohr M, Schrittwieser R, Simon T, Smietanski M, Stechemesser B, Timoney M, Chowbey P; International Endohernia Society (IEHS). Guidelines for laparoscopic treatment of ventral and incisional abdominal wall hernias (International Endohernia Society (IEHS)—Part 1. Surgical Endoscopy. 2014;28(1):2-29.  https://doi.org/10.1007/s00464-013-3170-6
  18. Fitzgibbons RJ Jr, Giobbie-Hurder A, Gibbs JO, Dunlop DD, Reda DJ, McCarthy M Jr, Neumayer LA, Barkun JS, Hoehn JL, Murphy JT, Sarosi GA Jr, Syme WC, Thompson JS, Wang J, Jonasson O. Watchful Waiting vs Repair of Inguinal Hernia in Minimally Symptomatic Men. JAMA. 2006;295(3):285.  https://doi.org/10.1001/jama.295.3.285
  19. Köckerling F, Bittner R, Kuthe A, Hukauf M, Mayer F, Fortelny R, Schug-Pass C. TEP or TAPP for recurrent inguinal hernia repair—register-based comparison of the outcome. Surgical Endoscopy. 2017;31(10):3872-3882. https://doi.org/10.1007/s00464-017-5416-1
  20. De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, Rizzo C. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Frontiers in Public Health. 2023;11:1166120. https://doi.org/10.3389/fpubh.2023.1166120
  21. Janssen BV, Kazemier G, Besselink MG. The use of ChatGPT and other large language models in surgical science. BJS Open. 2023;7(2):zrad032. https://doi.org/10.1093/bjsopen/zrad032
  22. Samaan JS, Yeo YH, Rajeev N, Hawley L, Abel S, Ng WH, Srinivasan N, Park J, Burch M, Watson R, Liran O, Samakar K. Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery. Obesity Surgery. 2023;33(6):1790-1796. https://doi.org/10.1007/s11695-023-06603-5
  23. Ayoub NF, Lee YJ, Grimm D, Balakrishnan K. Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient Instructions. JAMA Otolaryngology — Head & Neck Surgery. 2023;149(6):556.  https://doi.org/10.1001/jamaoto.2023.0704
  24. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A, Schulman J, Hilton J, Kelton F, Miller L, Simens M, Askell A, Welinder P, Christiano P, Leike J, Lowe R. Training language models to follow instructions with human feedback. Arxiv. March 4, 2022. Accessed August 11, 2023. https://arxiv.org/abs/2203.02155
  25. Haupt CE, Marks M. AI-Generated Medical Advice — GPT and Beyond. JAMA. 2023;329(16):1349. https://doi.org/10.1001/jama.2023.5321
  26. Jones E, Steinhardt J. Capturing Failures of Large Language Models via Human Cognitive Biases. Arxiv. February 24, 2022. Accessed August 18, 2023. https://arxiv.org/abs/2202.12299
  27. Kirchner JH. Cognitive Biases in Large Language Models. On Brains, Minds, And Their Possible Uses. September 25, 2021. Accessed August 12, 2023. https://universalprior.substack.com/p/cognitive-biases-in-large-language?utm_campaign=post&utm_medium=web
Contact the author
Email
Message
The publishing house "Media Sphere" does not provide treatment services, does not give recommendations on contacting medical institutions and specific specialists, on the prescription of medicines and dietary supplements.

Email Confirmation

An email was sent to test@gmail.com with a confirmation link. Follow the link from the letter to complete the registration on the site.

Email Confirmation

Contact us
If you have any questions, complaints or suggestions, please use this feedback form.
Email
Message
The publishing house "Media Sphere" does not provide treatment services, does not give recommendations on contacting medical institutions and specific specialists, on the prescription of medicines and dietary supplements.
Submit Application

You can become an author of one of our magazines, send a request and we will consider it in during the day.

Name
Phone
E-mail
Message
Login
Registration
E-mail
Password
Forgot Password?

Log in to the site using your account in one of the services

Password recovery
E-mail
Login
Register

We use cооkies to improve the performance of the site. By staying on our site, you agree to the terms of use of cооkies. To view our Privacy and Cookie Policy, please. click here.