• Login
Healthyduck
  • Home
  • News
  • Medicines
  • Gym
  • Fitness
  • Nutrition
  • Covid-19
No Result
View All Result
  • Home
  • News
  • Medicines
  • Gym
  • Fitness
  • Nutrition
  • Covid-19
No Result
View All Result
HealthNews
No Result
View All Result
Home Medicines

AI Bot ChatGPT Passes US Medical Licensing Exams

Steve Heldon by Steve Heldon
January 27, 2023
in Medicines
0
0
SHARES
0
VIEWS
Share on FacebookShare on Twitter

ChatGPT can pass parts of the US medical licensing exam, researchers have found, raising questions about whether the AI ​​chatbot could one day help write the exam or help students prepare for it.



Victor Tseng, MD, and his colleagues at Ansible Health, a company that manages mostly homebound patients with chronic lung disease, initially wanted to see whether ChatGPT could aggregate all the communications regarding these patients, which would allow Ansible to better coordinate care.

“Naturally, we wondered how ChatGPT might augment patient care,” Tseng, Ansible’s vice president and medical director, told Medscape. A group of volunteers at the company decided to test its capabilities by asking it multiple choice questions from the US Medical Licensing Examination (USMLE), given that so many of them had taken the medical licensing exam.

“The results were so shocking to us that we sprinted to turn it into a publication,” said Tseng. The results were published as a preprint on withRxiv. They were so impressed that they allowed ChatGPT to collaborate as a contributing author.

ChatGPT wrote the abstract and results sections “with minimal prompting and largely cosmetic adjustments from the human co-authors,” said Tseng. The bot also contributed large sections to the introduction and methods sections. The authors “frequently asked it to synthesize, simplify, and offer counterpoints to drafts in progress,” Tseng said. He likened it to how co-authors might interact over email. They decided they would not credit ChatGPT as an author, however.

The article has been accepted in the peer-reviewed journal PLOS Digital Health and will be published soon, Tseng told Medscape.

Alex Mechaber, MD, vice president of the USMLE Program at the National Board of Medical Examiners (NBME), said the organization is not surprised by the study’s results, “in part because the input material that’s used for ChatGPT is largely representative of medical knowledge .” AI is most likely to be successful with multiple-choice type questions, Mechaber told Medscape.

San Francisco–based OpenAI developed ChatGPT, a large language model. The tech giant Microsoft considers ChatGPT and OpenAI’s other applications so promising that it has already invested $3 billion and is reportedly poised to put another $10 billion into the company.

ChatGPT’s algorithms are “trained to predict the likelihood of a given sequence of words based on the context of the words that come before it.” Theoretically, it is “capable of generating novel sequences of words never observed previously by the model, but that represent plausible sequences based on natural human language,” according to Tseng and his co-authors.

Released to the public in November 2022, ChatGPT has been used to write everything from love poems to high school history papers to website editorial content. The bot draws on a data store that includes everything that has been uploaded to the internet through 2021.

Tseng and colleagues tested ChatGPT on hundreds of multiple-choice questions covered in the three steps of the USMLE exam.

For each step, the researchers prompted the chatbot in three ways. First, it was given a theoretical patient’s signs and symptoms and asked to pontificate on what might be the underlying cause or diagnosis.

Next, after ChatGPT was refreshed to eliminate potential bias from any retained information from the previous exercise, it was given the questions from the exam and asked to pick an answer. After again refreshing ChatGPT, the researchers asked it to “please explain why the correct answers are correct and why the incorrect answers are incorrect.”

The answers were reviewed and scored by three board-certified, licensed physicians.

For the open-ended format, ChatGPT’s accuracy for Step 1 ranged from 43% when “indeterminate” answers were included in the analysis to 68% when those responses were excluded. An indeterminate answer is one in which the chatbot either gave a response that was not available among the multiple choices that were presented or said it could not commit to an answer. For Step 2, the pass rate was 51%/58%, and for Step 3, it was 56%/62%.

When asked the questions verbatim, ChatGPT’s accuracy was 36/55% for Step 1, 57%/59% for Step 2CK, and 55%/61% for Step 3. When asked to justify its responses, its accuracy rate was 40%/ 62% for Step 1, 49%/51% for Step 2, and 60%/65% for Step 3.

The pass rate for students varies according to whether it’s a first exam or a repeat exam and whether the test-taker is from the United States or a different country. In 2021, for Step 1, the pass rate ranged from a low of 45% for repeaters to a high of 96%. For Step 2, the range was 62% to 99%, and for Step 3, the range was 62% to 98%.

“What’s fascinating is that in the Step 2 and 3, which are more clinically advanced, only around 10% of [ChatGPT’s] responses were indeterminate,” said Tseng.

Bot Not Tested on Crucial Parts of Exam

USMLE’s Mechaber noted that ChatGPT was only given a sampling of questions, not an actual practice test. And it did not attempt questions that use images or sounds or the case-based computer simulation studies administered in Step 3, he said.

Tseng suggests in his article that ChatGPT could potentially be used as a study aide for students preparing for the USMLE or to write questions for the exam.

“We’re thinking about that,” Mechaber said about its use as a study tool. But since ChatGPT still produces so many wrong answers, the technology is not likely “ready for prime time,” he said. As to whether ChatGPT could write test questions, the NBME has shown interest in “automated item generation,” he said.

“We’re investigating [ChatGPT] with excitement and curiosity” for its potential use in medicine, Mechaber said.

Chatbot Says USMLE Is Here to Stay

An NBME staff member decided to query ChatGPT about whether it was a threat to the USMLE. The bot said that while it is a “powerful tool for natural language processing,” it “is not a threat to the United States Medical Licensing Examination (USMLE).”

In a lengthy response, the bot added, “ChatGPT, while impressive in its ability to generate human-like text, is not specifically designed to test medical knowledge and is not a substitute for the rigorous training and education required to become a licensed physician. “

In addition, ChatGPT “does not have the ability to think critically or solve problems in the way that a human physician would,” it said.

The bot also brought up ethical considerations, noting that since AI models “are based on machine learning which can be biased, hence the results generated by the model may not be accurate and unbiased.

“ChatGPT is an impressive tool for natural language processing, but it is not a replacement for the specialized knowledge, critical thinking and ethical considerations that are essential for the practice of medicine,” it said. “The USMLE remains an important and valid way to evaluate the knowledge and abilities of aspiring physicians,” said the bot.

The study was conducted by volunteers and was not funded by any source. Tseng is a full-time employee of and writes test questions for U World, a USMLE test prep company.

Alicia Ault is a Saint Petersburg, Florida–based freelance journalist whose work has appeared in publications including JAMA and Smithsonian.com. You can find her on Twitter @aliciaault.

For more news, follow Medscape on Facebook, TwitterInstagram, and YouTube.

.

Tags: aesthetic medicineAIartificial intelligenceboardscosmetic dermatologycosmeticsExercisefacial rejuvenationFloridahealth and med techhealth and medical techhealth and medical technologyhealthcare and medical technologyinternetlungpeer reviewphysical activitystudentuglyUnited States Medical Licensing Examination (USMLE)usmle
Previous Post

Dr. Ajay Gopal named Stephen H. Petersdorf Endowed Chair in Cancer Care at UW Medicine

Next Post

UR Medicine plans to reopen Jan. 30

Steve Heldon

Steve Heldon

Next Post

UR Medicine plans to reopen Jan. 30

Discussion about this post

Recommended

Ten Tar Heel Gymnasts named WGCA Scholastic All-America

1 year ago

Azithromycin to Prevent Sepsis or Death in Women Planning a Vaginal Birth | NEJM – nejm.org

1 month ago

Advertisement

Don't Miss

DEA Issues Public Health Warning Over Dangerous New Drug – The Daily Beast

March 21, 2023

Best cheap fitness tracker 2023: All the top budget fitness trackers reviewed – Stuff

March 21, 2023

Why You Should Be Using Barbell Collars – Gear Patrol

March 21, 2023

Free Fitness Classes Around DC This Week: March 20-26 – Washingtonian

March 21, 2023
Healthyduck

We bring you the best Premium WordPress Themes that perfect for news, magazine, personal blog, etc. Check our landing page for details.

Follow us

Recent News

DEA Issues Public Health Warning Over Dangerous New Drug – The Daily Beast

March 21, 2023

Best cheap fitness tracker 2023: All the top budget fitness trackers reviewed – Stuff

March 21, 2023

Categories

  • Beauty
  • Diet & Fitness
  • Disease
  • Family&Pregnancy
  • Fitness
  • Gym
  • Health
  • Lifestyle
  • Medicines
  • Mental Health
  • News
  • Nutrition
  • Pregnancy
  • Weight Loss

Tags

apmediaapi Business Cancer Coronavirus covid COVID-19 Covid-19 pandemic dailymail Daily mail dcc diabetes Diseases and conditions Education Epidemics Exercise Fitness food general news Gymnastics Health Health care Hospital Infectious diseases Lifestyle Lung disease Medicine News nutrition omicron Omicron variant pandemic political Political / general news Politics Public health research Science SIN Sport THE EPOKTIDEN thread United States vaccine weight loss WSJ-PRO-WSJ.com
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 JNews - Premium WordPress news & magazine theme by Jegtheme.

No Result
View All Result
  • Home
  • News
  • Medicines
  • Gym
  • Fitness
  • Nutrition
  • Covid-19

© 2023 JNews - Premium WordPress news & magazine theme by Jegtheme.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
Go to mobile version