Speaker: Matthew O'Rourke
Amount of medical knowledge is increasing exponentially. While physicians don’t need to know everything, part of the problem is that they don’t know what they need to know. He started with a medical journal, it evolved and became more multi-media and interactive. However, to maintain relevance they needed to go beyond that format completely. Their needs assessment showed:
What MDs want:
Progressive learning
Make the most of my time
Make it rewarding for me
Learn it my own way
What MDs need:
To demonstrate clinical knowledge in training
To demonstrate clinical knowledge in exams
To maintain and grow clinical knowledge in practice
Current learning science:
Spaced repetition
Metacognition (do you know what you don’t know?)
Bite-sized learning
ILPs (individualized learning plans – a trend now in medical education)
Adaptive learning
Physicians are frustrated by the piecemeal approach of occasional required courses, the need for both continual and just-in-time learning was great.
Adaptive learning (as opposed to traditional):
Adapts to individual
Assesses first, then offers targeted content
Takes into consideration what learner already knows and focuses on knowledge gaps
Is self-directed, with follow-up on what the learner has trouble with
They created NEJM Knowledge Plus, a tool that either provides questions or lets admins create them. It is used mostly for CMEs and also preparation in taking the Board exams.
There is a self-assessment dashboard for the learner to see. The majority of their users not only access on phone, but don’t even use a desktop. They use the term “probes” instead of questions, since traditional multiple-choice questions are one way to do this, but not the only way.
When learner answers the question they have to also answer their confidence level in their answer - “I know it”, “I think so”, “not sure”, or “no clue”. So if they get it right, but without confidence they will see this kind of question again to make sure it wasn’t a lucky guess. Repeated questions are spaced out to help with mastery. It also “primes the pump” for the learner to think more about their response.
The Ebbinghaus Forgetting Curve shows how forgetting time shortens the more one is exposed to the knowledge.
For the content creators, the hardest part of the process is to provide the feedback on the questions – why the correct answer is correct, and why the other answers are incorrect. They are finding this also leads to a good click through rate on the additional resources.
They also have a “challenge us” with each question – a way for learner to give feedback to them to help them keep on top of information and keep it current. This also helps learner to more deeply think about the question to articulate why they think it might be inaccurate or unclear.
Metacognition
They show the learner how their confidence level aligned with the correct/incorrect answer. Doctors who have a high degree of confidence and are often wrong are the most dangerous, and need to have this self-awareness. Other than system error, over-confidence is one of the biggest causes of medical error.